http://www.visiocafe.com/index.htm
常常做投影片，需要話架構圖，是不是都找不到適合的網路元件圖呢，這裡有~~~~
有DELL, HP等等，而且是免費的，常常來看看吧。
星期二, 10月 20, 2009
星期一, 11月 17, 2008
星期一, 11月 03, 2008
[資訊] What is zero-copy?
http://stenlyho.blogspot.com/2008/08/zero-copy.html
http://www.linuxjournal.com/article/6345
http://en.wikipedia.org/wiki/Zero-copy
http://www.ibm.com/developerworks/library/j-zerocopy/index.html
什麼是zero-copy?
簡單講就是copy data不經過user的copy，直接由kernel端做掉，如果硬體支援gather，就不需要copy到socket buffer，直接append就可以了，少了copy的行為，減少cpu與memory使用，自然performance就好啦~~~~^^
為啥為看到zero-copy呢?
因為Google protocol buffer有用嚕~~~~^^
http://www.linuxjournal.com/article/6345
http://en.wikipedia.org/wiki/Zero-copy
http://www.ibm.com/developerworks/library/j-zerocopy/index.html
什麼是zero-copy?
簡單講就是copy data不經過user的copy，直接由kernel端做掉，如果硬體支援gather，就不需要copy到socket buffer，直接append就可以了，少了copy的行為，減少cpu與memory使用，自然performance就好啦~~~~^^
為啥為看到zero-copy呢?
因為Google protocol buffer有用嚕~~~~^^
星期五, 10月 24, 2008
[資訊] Windows live messenger 9 安裝在windows 2003
http://heresy.spaces.live.com/blog/cns!E0070FB8ECF9015F!5025.entry
http://space.uwants.com/html/64/1843664-181786.html
最近看到Windows live messenger 9.0 Beta，本來想說等正式版出來再來用好了
可是剛剛手養就安裝了，由於目前有一台 windows 2003 server，所以就查了一下怎麼安裝 windows live messenger 9.0 Beta
先到XP的機器安裝 windows live messenger 9.0 Beta，然後再到目錄
C:\Program Files\Common Files\Windows Live\.cache下面有好幾個子目錄
找到Messenger.msi與Contacts.msi，然後copy到 windows 2003 server就可以安裝了
可是我安裝完，在"新增移除程式"中倒是沒看到 windows live messenger 的安裝，到時候不知道怎麼移除說~~~~^^
不管反正就是安裝完成
另外會出現8100030d的錯誤，就參考第二個link的說明，不過我倒是覺得沒做，偶爾有問題，做了就比較不會出現錯誤。
好，安裝完成，好好玩 windows live messenger 9.0 Beta，感覺改版很多，雖然主要還是傳message~~~~
http://space.uwants.com/html/64/1843664-181786.html
最近看到Windows live messenger 9.0 Beta，本來想說等正式版出來再來用好了
可是剛剛手養就安裝了，由於目前有一台 windows 2003 server，所以就查了一下怎麼安裝 windows live messenger 9.0 Beta
先到XP的機器安裝 windows live messenger 9.0 Beta，然後再到目錄
C:\Program Files\Common Files\Windows Live\.cache下面有好幾個子目錄
找到Messenger.msi與Contacts.msi，然後copy到 windows 2003 server就可以安裝了
可是我安裝完，在"新增移除程式"中倒是沒看到 windows live messenger 的安裝，到時候不知道怎麼移除說~~~~^^
不管反正就是安裝完成
另外會出現8100030d的錯誤，就參考第二個link的說明，不過我倒是覺得沒做，偶爾有問題，做了就比較不會出現錯誤。
好，安裝完成，好好玩 windows live messenger 9.0 Beta，感覺改版很多，雖然主要還是傳message~~~~
星期二, 9月 23, 2008
OllyDbg 組譯程式
http://forum.slime.com.tw/thread175371.html
這是一個組譯程式，我記錄一下，有需要用的時候才不會忘記。
聽說阿碼科技用源碼檢測，我小小的猜測了一下
如果是我的話，我會這麼做
應該是找出組譯程式的行為吧，如果不是那就是在程式碼上的檢測，不過那就比較有爭議了
....^____^
當然講很簡單，其中還是有很多訣竅的
不過只要精神抓到了，接下來就是投入的精力有多少了
Update:
嗯，猜錯了，應該是針對JavaScript的物件做分析
Update:
嗯，猜錯了，請參考Link
這是一個組譯程式，我記錄一下，有需要用的時候才不會忘記。
聽說阿碼科技用源碼檢測，我小小的猜測了一下
如果是我的話，我會這麼做
應該是找出組譯程式的行為吧，如果不是那就是在程式碼上的檢測，不過那就比較有爭議了
....^____^
當然講很簡單，其中還是有很多訣竅的
不過只要精神抓到了，接下來就是投入的精力有多少了
Update:
嗯，猜錯了，應該是針對JavaScript的物件做分析
Update:
嗯，猜錯了，請參考Link
星期三, 7月 30, 2008
[Hadoop]Automatic namespace recovery from the secondary image.
太棒了，我找到secondary namenode如何recovery了~~~~
大家可以參考這個link
我節錄重點如下：
We had a real example of such failure on one of our clusters.
And we were able to reconstruct the namespace image from the secondary node using the following manual procedure, which might be useful for those who find themselves in the same type of trouble.
Manual recovery procedure from the secondary image.
1. Stop the cluster to make sure all data-nodes and *-tracker are down.
2. Select a node where you will run a new name-node, and set it up as usually for the name-node.
3. Format the new name-node.
4. cd/current
5. You will see file VERSION in there. You will need to provide namespaceID of the old cluster in it.
The old namespaceID could be obtained from one of the data-nodes
just copy it from/current/VERSION.namespaceID
6. rm/current/fsimage
7. scp:/destimage.tmp ./fsimage
8. Start the cluster. Upgrade is recommended, so that you could rollback if something goes wrong.
9. Run Hadoop fsck, and remove files with missing blocks if any.
Automatic recovery proposal.
The proposal consists has 2 parts.
1. The secondary node should store the latest check-pointed image file in compliance with the name-node storage directory structure. It is best if secondary node uses Storage class (or FSImage if code re-use makes sense here) in order to maintain the checkpoint directory.
This should provide that the checkpointed image is always ready to be read by a name-node if the directory is listed in its "dfs.name.dir" list.
2. The name-node should consider the configuration variable "fs.checkpoint.dir" as a possible location of the image available for read-only access during startup.
This means that if name-node finds all directories listed in "dfs.name.dir" unavailable or finds their images corrupted, then it should turn to the "fs.checkpoint.dir" directory and try to fetch the image from there. I think this should not be the default behavior but rather triggered by a name-node startup option, something like:
hadoop namenode -fromCheckpoint
So the name-node can start with the secondary image as long as the secondary node drive is mounted.
And the name-node will never attempt to write anything to this drive.
Added bonuses provided by this approach
* One can choose to restart failed name-node directly on the node where the secondary node ran.
This brings us a step closer to the hot standby.
* Replication of the image to NFS can be delegated to the secondary name-node if we will
support multiple entries in "fs.checkpoint.dir". This is of course if the administrator
chooses to accept outdated images in order to boost the name-node performance.
目前先用manual的方式處理，之後再看automatic recovery做的怎麼樣再說嚕。
------------------------------------------------------------------------------------
Update:
經過不斷的試驗，發現根本還沒做好，自己備份namenode的檔案還比較快，也確認可以完全回復HDFS....>_<
大家可以參考這個link
我節錄重點如下：
We had a real example of such failure on one of our clusters.
And we were able to reconstruct the namespace image from the secondary node using the following manual procedure, which might be useful for those who find themselves in the same type of trouble.
Manual recovery procedure from the secondary image.
1. Stop the cluster to make sure all data-nodes and *-tracker are down.
2. Select a node where you will run a new name-node, and set it up as usually for the name-node.
3. Format the new name-node.
4. cd
5. You will see file VERSION in there. You will need to provide namespaceID of the old cluster in it.
The old namespaceID could be obtained from one of the data-nodes
just copy it from
6. rm
7. scp
8. Start the cluster. Upgrade is recommended, so that you could rollback if something goes wrong.
9. Run Hadoop fsck, and remove files with missing blocks if any.
Automatic recovery proposal.
The proposal consists has 2 parts.
1. The secondary node should store the latest check-pointed image file in compliance with the name-node storage directory structure. It is best if secondary node uses Storage class (or FSImage if code re-use makes sense here) in order to maintain the checkpoint directory.
This should provide that the checkpointed image is always ready to be read by a name-node if the directory is listed in its "dfs.name.dir" list.
2. The name-node should consider the configuration variable "fs.checkpoint.dir" as a possible location of the image available for read-only access during startup.
This means that if name-node finds all directories listed in "dfs.name.dir" unavailable or finds their images corrupted, then it should turn to the "fs.checkpoint.dir" directory and try to fetch the image from there. I think this should not be the default behavior but rather triggered by a name-node startup option, something like:
hadoop namenode -fromCheckpoint
So the name-node can start with the secondary image as long as the secondary node drive is mounted.
And the name-node will never attempt to write anything to this drive.
Added bonuses provided by this approach
* One can choose to restart failed name-node directly on the node where the secondary node ran.
This brings us a step closer to the hot standby.
* Replication of the image to NFS can be delegated to the secondary name-node if we will
support multiple entries in "fs.checkpoint.dir". This is of course if the administrator
chooses to accept outdated images in order to boost the name-node performance.
目前先用manual的方式處理，之後再看automatic recovery做的怎麼樣再說嚕。
------------------------------------------------------------------------------------
Update:
經過不斷的試驗，發現根本還沒做好，自己備份namenode的檔案還比較快，也確認可以完全回復HDFS....>_<
星期二, 7月 29, 2008
Hadoop 之 Configuration from Installation
Separating Configuration from Installation
In the example described above, the configuration files used by the Hadoop cluster all lie in the Hadoop installation. This can become cumbersome when upgrading to a new release since all custom config has to be re-created in the new installation. It is possible to separate the config from the install. To do so, select a directory to house Hadoop configuration (let's say /foo/bar/hadoop-config. Copy the hadoop-site.xml, slaves and hadoop-env.sh files to this directory. You can either set the HADOOP_CONF_DIR environment variable to refer to this directory or pass it directly to the Hadoop scripts with the --config option. In this case, the cluster start and stop commands specified in the above two sub-sections become
% $HADOOP_INSTALL/hadoop/bin/start-all.sh --config /foo/bar/hadoop-config and
% $HADOOP_INSTALL/hadoop/bin/stop-all.sh --config /foo/bar/hadoop-config.
Only the absolute path to the config directory should be passed to the scripts.
這個段落節錄於Hadoop office web site，可以點這個link連過去。
這個還蠻有用的，可以把Hadoop installation與configuration分開，記錄一下。
In the example described above, the configuration files used by the Hadoop cluster all lie in the Hadoop installation. This can become cumbersome when upgrading to a new release since all custom config has to be re-created in the new installation. It is possible to separate the config from the install. To do so, select a directory to house Hadoop configuration (let's say /foo/bar/hadoop-config. Copy the hadoop-site.xml, slaves and hadoop-env.sh files to this directory. You can either set the HADOOP_CONF_DIR environment variable to refer to this directory or pass it directly to the Hadoop scripts with the --config option. In this case, the cluster start and stop commands specified in the above two sub-sections become
% $HADOOP_INSTALL/hadoop/bin/start-all.sh --config /foo/bar/hadoop-config and
% $HADOOP_INSTALL/hadoop/bin/stop-all.sh --config /foo/bar/hadoop-config.
Only the absolute path to the config directory should be passed to the scripts.
這個段落節錄於Hadoop office web site，可以點這個link連過去。
這個還蠻有用的，可以把Hadoop installation與configuration分開，記錄一下。
Hadoop的secondary namenode設定
這個link
有人回應說要設定secondary namenode需要設定hadoop-default.xml裡面的dfs.secondary.http.address
這到很有趣，設定secondary namenode到底有哪些步驟呢?
等我搞懂了，在看要不要PO出來~~~~^^
-------------------------------------------------------------------------------
Update:
是設定hadoop-site.xml中的dfs.secondary.http.address
還有fs.checkpoint.dir即可
至於fs.checkpoint.period與fs.checkpoint.size是看個人設定
但其實一點效果也沒有，自己備份還比較快，只是要寫script罷了...
有人回應說要設定secondary namenode需要設定hadoop-default.xml裡面的dfs.secondary.http.address
這到很有趣，設定secondary namenode到底有哪些步驟呢?
等我搞懂了，在看要不要PO出來~~~~^^
-------------------------------------------------------------------------------
Update:
是設定hadoop-site.xml中的dfs.secondary.http.address
還有fs.checkpoint.dir即可
至於fs.checkpoint.period與fs.checkpoint.size是看個人設定
但其實一點效果也沒有，自己備份還比較快，只是要寫script罷了...
星期五, 7月 25, 2008
Hadoop之我見
Fackbook 正式使用Hadoop
http://www.facebook.com/note.php?note_id=16121578919
這代表什麼意思呢?....
資料越來越多，傳統的資料統計方式已經過時了
所以才走進雲端計算....
但其實可以發現的是要夠有錢，丟1000台機器做運算，
假設一台機器1U是10萬好了，就是10000萬 = 十億，丟十億做資料統計，相信沒幾間公司做的到(yahoo就做了)。
如果機器差一點只要3萬好了，就是3000萬，我想也沒有公司願意做。
這些都不包含 maintain and programming 的工程師。
所以Google現在就要走向租computing power，不想花錢沒關係，跟我租就好了，一次根據你使用的CPU使用率，或是硬碟的使用率來收錢，屌吧。
上次去參加 Google developer day 2008 ，我就非常有這種感覺了，而且誰都可以做，不限公司，不限個人，目前最大的問題就是資料如果牽扯到保密規定呢?....
其實這個問題好解決，就客製化就好了，公司只要說好要幾台，就幫你搞定一個單獨的區域，給你登入的方式，你自己用，要用多久就收費多少，不跟別人放在同樣的DFS就好了。
不過根據Google的文化，他一定覺得賺全球人的錢，一定比只賺企業的錢來得多。
其實還有另外的一種方式，就是我在各地區建立Data center，幫你建立雲端運算(cloud computing)，我收maintain的錢，你只要用多久付多久，只是這種方式得看該地區是不是真的需求。
以台灣來看，根本就沒有需求。以電信業者而言，電信業的資料超大也超多，可是沒用，因為沒有revenue，雖然可以根據行為分析去做行銷，但是誰也不敢保證做了，賺的revenue足夠付成本還有剩，其實根據某電信只要多算五秒鐘就幾億的謠言，我相信其實是可期的，賺錢的方向越來越不像之前，以手機為走向了。以前只要 Sony Ericsson 或是 Nokia 有出新手機，電信業者有搭配門號就可以了，可是現在手機業者也不能這樣獲利了，也開始走加值功能，所以HTC, iphone, Gphone都是朝這個方向走了。
這樣代表什麼意思呢?就是手機業成熟了，不再是打打電話，而是更多的服務更多的功能，但這些都需要結合content delivery，所以手機業的戰場變成他能 deliver 多少的服務或功能，而電信業者不再是靠這手機增加門號，門號已經飽和了，重要的是通話秒數，資料傳遞量。
哈，忽然講一堆，真是怪怪的，其實只是要表達的是戰場改變，生活習慣也開始改變，都是因為雲端運算(cloud computing)越來越貼近人群，也因為資料量的龐大，開始不一樣的改變，以工程師的角度 programming 的方式開始不一樣，機器的 maintain也開始不一樣，以產業經營者的角度來看，獲利方式也開是不一樣。
當然這一切還需要時間，多久，我也不知道，但我想以Google這麼強大的影響力....
應該在不久的將來。
其實我也在想另外的一個議題，全球暖化跟太多台電腦有關係嗎??....
http://www.facebook.com/note.php?note_id=16121578919
這代表什麼意思呢?....
資料越來越多，傳統的資料統計方式已經過時了
所以才走進雲端計算....
但其實可以發現的是要夠有錢，丟1000台機器做運算，
假設一台機器1U是10萬好了，就是10000萬 = 十億，丟十億做資料統計，相信沒幾間公司做的到(yahoo就做了)。
如果機器差一點只要3萬好了，就是3000萬，我想也沒有公司願意做。
這些都不包含 maintain and programming 的工程師。
所以Google現在就要走向租computing power，不想花錢沒關係，跟我租就好了，一次根據你使用的CPU使用率，或是硬碟的使用率來收錢，屌吧。
上次去參加 Google developer day 2008 ，我就非常有這種感覺了，而且誰都可以做，不限公司，不限個人，目前最大的問題就是資料如果牽扯到保密規定呢?....
其實這個問題好解決，就客製化就好了，公司只要說好要幾台，就幫你搞定一個單獨的區域，給你登入的方式，你自己用，要用多久就收費多少，不跟別人放在同樣的DFS就好了。
不過根據Google的文化，他一定覺得賺全球人的錢，一定比只賺企業的錢來得多。
其實還有另外的一種方式，就是我在各地區建立Data center，幫你建立雲端運算(cloud computing)，我收maintain的錢，你只要用多久付多久，只是這種方式得看該地區是不是真的需求。
以台灣來看，根本就沒有需求。以電信業者而言，電信業的資料超大也超多，可是沒用，因為沒有revenue，雖然可以根據行為分析去做行銷，但是誰也不敢保證做了，賺的revenue足夠付成本還有剩，其實根據某電信只要多算五秒鐘就幾億的謠言，我相信其實是可期的，賺錢的方向越來越不像之前，以手機為走向了。以前只要 Sony Ericsson 或是 Nokia 有出新手機，電信業者有搭配門號就可以了，可是現在手機業者也不能這樣獲利了，也開始走加值功能，所以HTC, iphone, Gphone都是朝這個方向走了。
這樣代表什麼意思呢?就是手機業成熟了，不再是打打電話，而是更多的服務更多的功能，但這些都需要結合content delivery，所以手機業的戰場變成他能 deliver 多少的服務或功能，而電信業者不再是靠這手機增加門號，門號已經飽和了，重要的是通話秒數，資料傳遞量。
哈，忽然講一堆，真是怪怪的，其實只是要表達的是戰場改變，生活習慣也開始改變，都是因為雲端運算(cloud computing)越來越貼近人群，也因為資料量的龐大，開始不一樣的改變，以工程師的角度 programming 的方式開始不一樣，機器的 maintain也開始不一樣，以產業經營者的角度來看，獲利方式也開是不一樣。
當然這一切還需要時間，多久，我也不知道，但我想以Google這麼強大的影響力....
應該在不久的將來。
其實我也在想另外的一個議題，全球暖化跟太多台電腦有關係嗎??....
星期三, 7月 23, 2008
Linux grep tab的用法
寫一下以免又忘了
如果data是
wwwwww
grep [[:blank:]]w[[:blank:]]
才會抓到w
記得要雙括號!!
下面還有一些使用方式
[:alnum:] 字母數字符號
[:alpha:] 字母
[:ascii:] ASCII符號
[:blank:] 空格或是Tab
[:cntrl:] ASCII碼控制符號
[:digit:] 數字
[:graph:] 非控制,非空格符號
[:lower:] 小寫字母
[:print:] 可打印符號
[:punct:] 邊點符號
[:space:] 空白符號,包括垂直Tab
[:upper:] 大寫符號
[:xdigit:] 十六進制數字
^ 在一行的開始
$ 在一行的結尾
. 任意一個單一字元
如果data是
www
grep [[:blank:]]w[[:blank:]]
才會抓到
記得要雙括號!!
下面還有一些使用方式
[:alnum:] 字母數字符號
[:alpha:] 字母
[:ascii:] ASCII符號
[:blank:] 空格或是Tab
[:cntrl:] ASCII碼控制符號
[:digit:] 數字
[:graph:] 非控制,非空格符號
[:lower:] 小寫字母
[:print:] 可打印符號
[:punct:] 邊點符號
[:space:] 空白符號,包括垂直Tab
[:upper:] 大寫符號
[:xdigit:] 十六進制數字
^ 在一行的開始
$ 在一行的結尾
. 任意一個單一字元
星期四, 7月 17, 2008
Hadoop-How many reduces?
How Many Reduces?
The right number of reduces seems to be 0.95 or 1.75 multiplied by ( * mapred.tasktracker.reduce.tasks.maximum).
With 0.95 all of the reduces can launch immediately and start transfering map outputs as the maps finish. With 1.75 the faster nodes will finish their first round of reduces and launch a second wave of reduces doing a much better job of load balancing.
Increasing the number of reduces increases the framework overhead, but increases load balancing and lowers the cost of failures.
The scaling factors above are slightly less than whole numbers to reserve a few reduce slots in the framework for speculative-tasks and failed tasks.
看不是很懂，猜一下好了，是指reducer跟data node與maximum reduce task有正面關係....
所以越多會增加framework的overhead，但可以增加load balancing和lowers the cost of failures.
The right number of reduces seems to be 0.95 or 1.75 multiplied by (
With 0.95 all of the reduces can launch immediately and start transfering map outputs as the maps finish. With 1.75 the faster nodes will finish their first round of reduces and launch a second wave of reduces doing a much better job of load balancing.
Increasing the number of reduces increases the framework overhead, but increases load balancing and lowers the cost of failures.
The scaling factors above are slightly less than whole numbers to reserve a few reduce slots in the framework for speculative-tasks and failed tasks.
看不是很懂，猜一下好了，是指reducer跟data node與maximum reduce task有正面關係....
所以越多會增加framework的overhead，但可以增加load balancing和lowers the cost of failures.
星期三, 7月 16, 2008
Hadoop map 設定
How Many Maps?
The number of maps is usually driven by the total size of the inputs, that is, the total number of blocks of the input files.
The right level of parallelism for maps seems to be around 10-100 maps per-node, although it has been set up to 300 maps for very cpu-light map tasks. Task setup takes awhile, so it is best if the maps take at least a minute to execute.
Thus, if you expect 10TB of input data and have a blocksize of 128MB, you'll end up with 82,000 maps, unless setNumMapTasks(int) (which only provides a hint to the framework) is used to set it even higher.
所以請將你的total data size / block size = map number，下台一鞠躬
The number of maps is usually driven by the total size of the inputs, that is, the total number of blocks of the input files.
The right level of parallelism for maps seems to be around 10-100 maps per-node, although it has been set up to 300 maps for very cpu-light map tasks. Task setup takes awhile, so it is best if the maps take at least a minute to execute.
Thus, if you expect 10TB of input data and have a blocksize of 128MB, you'll end up with 82,000 maps, unless setNumMapTasks(int) (which only provides a hint to the framework) is used to set it even higher.
所以請將你的total data size / block size = map number，下台一鞠躬
Hadoop system
最近上Hadoop的官網看到這段話
The client framework will implicitly associate the user identity with the connection to the name node, reducing the need for changes to the existing client API. It has always been the case that when one operation on a file succeeds, the operation might fail when repeated because the file, or some directory on the path, no longer exists. For instance, when the client first begins reading a file, it makes a first request to the name node to discover the location of the first blocks of the file. A second request made to find additional blocks may fail. On the other hand, deleting a file does not revoke access by a client that already knows the blocks of the file.
我寫中文比較能表示意思，表示當有人reading a file，同時有人做刪除或修改的動作，reading會錯誤。
表示說Hadoop並未做任何lock的動作....所以如果有read file from HDFS，記得要做Error handle。
The client framework will implicitly associate the user identity with the connection to the name node, reducing the need for changes to the existing client API. It has always been the case that when one operation on a file succeeds, the operation might fail when repeated because the file, or some directory on the path, no longer exists. For instance, when the client first begins reading a file, it makes a first request to the name node to discover the location of the first blocks of the file. A second request made to find additional blocks may fail. On the other hand, deleting a file does not revoke access by a client that already knows the blocks of the file.
我寫中文比較能表示意思，表示當有人reading a file，同時有人做刪除或修改的動作，reading會錯誤。
表示說Hadoop並未做任何lock的動作....所以如果有read file from HDFS，記得要做Error handle。
星期五, 5月 02, 2008
Hadoop能做什麼
最近在survey hadoop，發現能做的事很少，因為key-value只能一個值
一個值就很單調了，如果要複雜的計算根本不可能....
所以只能做基本的加減乘除....目前就我所知是這樣，不知道有沒有更好的應用
但是如果算好幾次來達成到是可行，所以function programming非常重要....
必須要自己先做方程式分解，先把需要計算的先算好，然後再組合....
所以想好公式，思考好key是什麼，分次計算，在集合起來。
一個值就很單調了，如果要複雜的計算根本不可能....
所以只能做基本的加減乘除....目前就我所知是這樣，不知道有沒有更好的應用
但是如果算好幾次來達成到是可行，所以function programming非常重要....
必須要自己先做方程式分解，先把需要計算的先算好，然後再組合....
所以想好公式，思考好key是什麼，分次計算，在集合起來。
星期四, 4月 24, 2008
Data mining
要做data mining的要點
1.先把資料流串好，能每天處理完每天的量
2.然後給予足夠的資訊來用在verify process與training process
3.處理每天的資料決定加入training sample的條件
4.確認model的正確性
5.重複1
The points to do data mining
1.Prepare the data source and finish processing the quantity of daily data.
2.Give the enough information to verify and training process.
3.Process the daily data and determine the criteria to add to training samples.
4.Verify the detection and accuracy rate of model.
5.repeat 1.
1.先把資料流串好，能每天處理完每天的量
2.然後給予足夠的資訊來用在verify process與training process
3.處理每天的資料決定加入training sample的條件
4.確認model的正確性
5.重複1
The points to do data mining
1.Prepare the data source and finish processing the quantity of daily data.
2.Give the enough information to verify and training process.
3.Process the daily data and determine the criteria to add to training samples.
4.Verify the detection and accuracy rate of model.
5.repeat 1.
訂閱：
文章 (Atom)
訂閱
發表文章
所有留言