Showing posts with label python. Show all posts
Showing posts with label python. Show all posts
Thursday, August 25, 2016
Python: How to Trace Source Code
Based on my previous Blog
http://gogosatellite.blogspot.tw/2016/05/the-best-and-simple-way-to-python-vim.html
After Installation,
We might use Ctrl+C, Ctrl +g (long press Ctrl) to trace code deeper.
and Press Ctrl+o to go back to original code location.
Great Tool for tracing source code tool.
Monday, July 4, 2016
SVM演算法於python的使用
前言
在很久以前，我是做演算法的，我记得，我最后一个接触的演算法就是SVM。 但那时只到听过这个演算法而已。 几年后，竟然有机会接触到演算法，竟然还是SVM。这过程真有趣。 以前总是想实现算法，现在python实在太方便了，lib呼叫就什么事情都解决了。
这也是当初我学Golang，最后转回python的原因。python的lib太强大了。
也感谢，聪明又美丽的北一女学妹，给我以下的blog，加速我的学习。
http://yourgene.pixnet.net/blog/post/115169530
SVM用法
from sklearn import svm
X = [[0, 0], [1, 1],[2, 2]]
y = [0, 1, 2]
clf = svm.SVC()
clf.fit(X, y)
print clf.predict([[5., 5.]])X為多維的training data，y為label。
One-Class SVM
One-Class
http://scikit-learn.org/stable/autoexamples/svm/plotoneclass.html
What's the meaning of nu
http://qiita.com/SE96UoC5AfUt7uY/items/88006646179adf68cb95
One-Class SVM，异常判定。 有的时候，我们有大量资料，我们只想知道，我们现在的资料是否为异常。 我们可以用以下的方式，实现。
from sklearn import svm
X = [[0, 0], [1, 1],[2, 2]]
y = [0, 1, 2]
clf = svm.OneClassSVM(nu=0.1, kernel="rbf", gamma=0.1)
clf.fit(X)
print clf.predict([[3, 3]])return值-1為異常。
Sunday, May 29, 2016
Introduce Python Classmethod and Staticmethod
Normal Method
An simple way to use class in file cst_tuto.py
class tt(object):
def __init__(self):
print 'into init'
self.a = 10
def normethod(self):
print 'in normal method'
return 'L in normethod'Adding the following lines at the end of the file, and execute it.
ttt = tt()
print ttt.normethod()
You will get this result
root@ubuntu:# python cst_tuto.py
into init
in normal method
L in normethodSo simple, you have to declaim the class as a instance, ttt = tt().
But, is There a simple way to call normethod just like a class ? just like
tt.normethod()The result is
root@ubuntu:# python cst_tuto.py
Traceback (most recent call last):
File "cst_tuto.py", line 12, in <module>
print tt.normethod()
TypeError: unbound method normethod() must be called with tt instance as first argument (got nothing instead)It's not working.
Same question, we now ask again is there a simpole way to call normethod just like a class ?
Yes, we can use staticmethod or classmethod. And I will show you what is the difference between them.
Staticmethod
Now we can modify cst_tuto.py as the followed.
class tt(object):
def __init__(self):
print 'into init'
self.a = 10
@staticmethod
def stamethod():
print 'in static'
return 'L in static'
def normethod(self):
print 'in normal method'
return 'L in normethod'We now execute it
ttt = tt()
## it will access __init__
print ttt.normethod()
print '-------------------'
print tt.stamethod()The result is
root@ubuntu:# python cst_tuto.py
into init
in normal method
L in normethod
-------------------
in static
L in staticYes, the result is satisfied our expectation that staticmethod can directly access the class, and no instance needed anymore.
ClassMethod
Based on the previous used file cst_tuto.py, we add classmethod in it.
class tt(object):
def __init__(self):
print 'into init'
self.a = 10
@staticmethod
def stamethod():
print 'in static'
return 'L in static'
@classmethod
def klamethod(cls):
print 'in klassmethod'
print cls
return 'L in klamethod'
def testmethod(self):
print 'in the test method'
return 'L in testmethod'
def normethod(self):
print 'in normal method'
return 'L in normethod'In the end of this file, and execute this python file.
print tt.klamethod()The result is
root@ubuntu:# python cst_tuto.py
in klassmethod
<class '__main__.tt'>
L in klamethodOf course it is satisfied our expectation again.
But what is cls. This is the only different between staticmethod and classmethod.
difference between staticmethod and classmethod
The staticmethod is a class that one can access it as a class not a instance. The classmethod is the same with staticmethod on this point, but classmethod can access the whole class tt by using cls, that staticmethod cannot do it. Staticmethod will be isolated and cannot contact to the other class.
The is an example that how the classmethod access the tt class. The hint is cls contains cls.__name__=tt that will help us to declaim the tt class.
class tt(object):
def __init__(self):
print 'into init'
self.a = 10
@staticmethod
def stamethod():
print 'in static'
return 'L in static'
@classmethod
def klamethod(cls):
print 'in klassmethod'
ttt = cls()
print ttt.testmethod()
return 'L in klamethod'
def testmethod(self):
print 'in the test method'
return 'L in testmethod'
def normethod(self):
print 'in normal method'
return 'L in normethod'
print tt.klamethod()
Adding ttt=cls() that is equivlent to ttt=tt() but don't forget classmethod only have one way to connect to outside world that is cls.
The result is
root@ubuntu:~# python cst_tuto.py
in klassmethod
into init
in the test method
L in testmethod
L in klamethodYes, we can access testmethod in the tt class. It's perfect.
The other question is
how to access classmethod from a normal method
def normethod(self):
# how to access klamethod
tt.klamethod()
print 'in normal method'
return 'L in normethod'You can access it just like the way to access classmethod.
how to access staticmethod from a classmethod
@classmethod
def klamethod(cls):
print 'in klassmethod'
print cls
print 'use stamethod in klamethod'
print cls.stamethod()In class method you can access staitcmethod without declaim ttt=cls(), but using cls.stamethod(). Just like to access a staticmethod. But you cannot get init attibute, if you don't decliam it as a instance.
How to import them and the behavior of the staticmehtod and classmethod
Adding the file ii_tuto.py, but don't forget to create the file init.py.
from cst import tt as yy
yy.stamethod()
yy.klamethod()
s = yy()
s.testmethod()
yy.normethod()We don't need to explain it too much, since it's similar to our explanation before.
More About Classmethod
As mentioned before, we emphisis the tt.__name__ is quite important concept using classmethod.
class Hero:
@staticmethod
def say_hello():
print("Helllo...")
@classmethod
def say_class_hello(cls):
if(cls.__name__=="HeroSon"):
print("Hi Kido")
elif(cls.__name__=="HeroDaughter"):
cls.class1 = HeroDaughter()
cls.class1.say_daughter_hello()
print("Hi Princess")
class HeroSon(Hero):
def say_son_hello(self):
print("test hello")
class HeroDaughter(Hero):
def say_daughter_hello(self):
print("test hello daughter")
testson = HeroSon()
testson.say_class_hello()
testson.say_hello()
testdaughter = HeroDaughter()
testdaughter.say_class_hello()
testdaughter.say_hello()The result is
"""
Hi Kido
Helllo...
Hi Princess
Helllo...
"""We inherent Class Hero and as a base class, and we can implement method depends on the input type in base class.
In the method of say_class_hello can identify which subclass calling and execute different function programing by ourself.
More About This Example
Furthermore Adding the following line to the beginning of the cst_tuto.py.
print 'haha'
class tt(object):
def __init__(self):
print 'into init'
self.a = 10
@staticmethod
def stamethod():
.
.
.Execute ii_tuto.py again
root@ubuntu:~/lab/python/classmethod# python ii_tuto.py
haha
in static
in klassmethodYou will see the result, when import cst the claim outside the class will be execute. Be careful about that, it will affect the performace alot, so clean it.
Thursday, May 19, 2016
Python Pecan Web Framework Tutorial (3) -- adding oslo-log
Python Pecan Web Framework Tutorial (3) -- adding oslo-log
我們繼續討論Pecan Web Framework未完成的部分-Oslo.log。在之前我們已經討論過oslo.confg如何跟pecan 結合
http://gogosatellite.blogspot.tw/2016/05/python-pecan-web-framework-tutorial-2.html
除此之外，我們也介紹了一個範例，如何使用Oslo-log。
http://gogosatellite.blogspot.tw/2016/05/oslo-log-tutorial.html
我們整合pecan，oslo-config，與oslo-log，程式部分放在我的Github中
https://github.com/jonahwu/lab/tree/master/pecan/pecantest/testprojectconfiglog
首先，我們先列出目錄結構
root@ubuntu:~/lab/pecan/pecan_test/test_project_config_log# tree
.
├── api.conf
├── api.log
├── api_paste.ini
├── api_paste.ini_bak
├── api.py
├── app_eventlet.py
├── app.py
├── app.pyc
├── app.wsgi
├── common
│ ├── _i18n.py
│ ├── i18n.py
│ ├── _i18n.pyc
│ ├── i18n.pyc
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── local.py
│ ├── local.pyc
│ ├── oslolog.py
│ └── oslolog.pyc
├── config.py
├── config.pyc
├── dist
│ └── test_project-0.1-py2.7.egg
├── healthcheck.py
├── healthcheck.pyc
├── hooks.py
├── hooks.pyc
├── __init__.py
├── __init__.pyc
├── MANIFEST.in
├── readme
├── service.py
├── service.pyc
├── setup.cfg
├── setup.py
├── test_project
│ ├── app.py
│ ├── app.pyc
│ ├── controllers
│ │ ├── __init__.py
│ │ ├── __init__.pyc
│ │ ├── root.py
│ │ ├── root.py_bak
│ │ ├── root.pyc
│ │ └── v2
│ │ ├── __init__.py
│ │ ├── __init__.pyc
│ │ ├── meters.py
│ │ ├── meters.pyc
│ │ ├── query2.py
│ │ ├── query2.pyc
│ │ ├── query.py
│ │ ├── query.pyc
│ │ ├── root.py
│ │ └── root.pyc
│ ├── __init__.py
│ ├── __init__.pyc重點我們先放在common目錄裡，這裏提供了所有需要的oslo-log的代碼，讀者只要將這目錄放到自己的檔案目錄中就可以使用了。 這部分，在我之前的Blog就有提到了
http://gogosatellite.blogspot.tw/2016/05/oslo-log-tutorial.html
加入oslo-log的相關程式於service.py
from oslo_config import cfg
import sys
import common.oslolog as logging
def prepare_service(argv=None):
if argv is None:
argv = sys.argv
cfg.CONF(argv[1:], project='app', validate_default_values=True)
CONF=cfg.CONF
DOMAIN = "app"
logging.setup(CONF, DOMAIN)大家可以比一下，與之前我的Blog中討論config的部分就知道其實改變並不大。
接下來就是，真實程式運作的部分，包含app.py與controller中程式的改變，以controller/v2/query.py為例 其中app.py只需要修改，以下三行即可
from common import oslolog as logging
from common.i18n import _
.
.
LOG = logging.getLogger(__name__)而controller中只需要加入兩行即可，以controller/v2/query.py為例
import common.oslolog as logging
LOG = logging.getLogger(__name__)在GET中我們修改為
@pecan.expose()
def get(self):
print 'show threaded we used %s'%CONF.api.threaded
print 'return query'
LOG.error('return query from logger')已經大功告成了，我們來執行看看
./api.py --debug --config-file /root/pecan_test/test_project/api.conf --log-file /root/pecan_test/test_project/api.log請注意這裏的目錄跟我們在github上所定義的目錄是不同的，讀者可隨之修正。
request為
curl http://localhost:8080/v2/query我們得到結果
Croot@ubuntu:~/pecan_test/test_project# cat api.log
2016-05-17 07:42:52.601 18175 ERROR app [-] -------------------api paste file /root/pecan_test/test_project/api_paste.ini
2016-05-17 07:42:52.616 18175 INFO werkzeug [-] * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
2016-05-17 07:42:54.203 18175 ERROR test_project.controllers.v2.query [-] return query from logger
2016-05-17 07:42:54.205 18175 INFO wsgi [-] 127.0.0.1 - - [17/May/2016:07:42:54 +0800] "GET /v2/query HTTP/1.1" 200 9 "-" "curl/7.35.0"
2016-05-17 07:42:54.205 18175 INFO werkzeug [-] 127.0.0.1 - - [17/May/2016 07:42:54] "GET /v2/query HTTP/1.1" 200 -我們得到結果return query from logger, oslo-log已經成功放入了。
一個application，config與log是不可或缺的，我們跟pecan整合起來，提供給讀者一個更容易使用的方法，加速開發的時間。
OpenStack oslo log Tutorial
Oslo-log Tutorial
這部分的代碼已經放到我的github上了
https://github.com/jonahwu/lab/tree/master/oslolog
另外這篇文章的內容，跟我之前一篇，oslo-config的文章有很大的關係，因為oslo-log相依於oslo-config。 因此我把oslo-config的文章也貼出來方便讀者查閱，並且我們也基於上篇oslo-config文章內容進行對oslo-log的程式添加，如下
http://gogosatellite.blogspot.tw/2016/05/openstack-oslo-config-tutorial.html
我們的作法基本上是從OpenStack Ceilometer中學習而來。
首先我們看一下目錄架構。
root@ubuntu:~/oslolog# tree
.
├── app.conf
├── app.log
├── app.py
├── app.py_bak
├── common
│ ├── _i18n.py
│ ├── i18n.py
│ ├── _i18n.pyc
│ ├── i18n.pyc
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── local.py
│ ├── local.pyc
│ ├── oslolog.py
│ └── oslolog.pyc
├── con.py
├── con.pyc
├── __init__.py
├── service.py
├── service.pyc記得，在任何目錄下，請增加__init__.py這個檔案，Python會將有這個檔案的目錄解析為Python目錄，因此
才可以使用from common.local import xx類似這樣的語法，如果沒有__init__.py會報Import Error的錯誤。
我想這是每個人都知道，但卻又容易犯的錯。
要使用oslo-log的機制，需要安裝非常多的oslo相關套件，以下為相關的套件，相對於oslo-config複雜多了。
root@ubuntu:~/oslolog# pip freeze|grep oslo
oslo.config==1.9.4.dev6
oslo.context==2.3.0
oslo.i18n==3.6.0
oslo.log==1.0.1.dev10
oslo.serialization==1.4.1.dev3
oslo.utils==3.10.0同樣的程式碼也相對複雜許多，因此我們把所有相關的程式碼放到common目錄裡，並將需要寫程式的部分分開，讓可用性與可讀性提升到最高。 讀者僅需要下載common的目錄並放置於相對應的位置即可使用。
我們首先來看看主要目錄發生了什麼事，首先，app.py增加了
from common import oslolog as logging
from common.i18n import _
.
.
.
LOG = logging.getLogger(__name__)所有的initial 我們放到service.py
from oslo_config import cfg
import sys
import common.oslolog as logging
def prepare_service(argv=None):
if argv is None:
argv = sys.argv
cfg.CONF(argv[1:], project='satellite', validate_default_values=True)
CONF=cfg.CONF
DOMAIN = "satellite"
logging.setup(CONF, DOMAIN)這部分跟之前我的oslo-config的文章比起來多了logging的設定，另外值得說明的是，要使用oslo-log需要搭配oslo-config才行。 否則無法寫到log file中，這部分，在我自己的測試中是這樣的，有沒有別的方法不依賴oslo-config我就不清楚了。
我們再看看，在其他檔案中，或其他Class中該怎麼使用oslo-log，我們看一下con.py這個檔案。
from oslo_config import cfg
import common.oslolog as logging
ALARM_API_OPTS = [
cfg.BoolOpt('record_history',
default=False,
help='Record alarm change events.'
),
cfg.StrOpt('alarm_sec',
default=0,
help='Record alarm change events.'
),
]
CONF = cfg.CONF
CONF.register_opts(ALARM_API_OPTS, group='alarm')
LOG = logging.getLogger(__name__)
class alm(object):
def __init__(self):
user_record_history = CONF.alarm.record_history
LOG.info('user record %s'%user_record_history)
LOG.info(CONF.alarm.alarm_sec)其實針對oslo-log的部分，我們只增加了兩行，
import common.oslolog as logging
LOG = logging.getLogger(__name__)原本相當複雜的設定，透過import common後變得非常容易，只需要兩行即可。
最後我們來看一下，該怎麼樣執行此程式檔，如下
./app.py --debug --config-file /root/oslolog/app.conf --log-file /root/oslolog/app.log其中--log-file是必須透過service.py內的CONF設定得來的，因此，這照成了oslo-log相依於oslo-config。
這時候，我們多了--log-file我們可以檢查一下，是否log的部分已經記載到此檔案了。
root@ubuntu:~/oslolog# cat app.log
2016-05-17 05:57:52.979 16960 ERROR __main__ [-] True
2016-05-17 05:57:52.979 16960 ERROR __main__ [-] ["['Magic'", "'Jessica'", "'Peter']"]
2016-05-17 05:57:52.980 16960 INFO con [-] user record True
2016-05-17 05:57:52.980 16960 INFO con [-] 20Wednesday, May 18, 2016
Python Pecan Web Framework Tutorial (2) -- adding hook and oslo-config
Introduction
這段代碼我放到我的github裡
https://github.com/jonahwu/lab/tree/master/pecan/pecantest/testproject_config
這篇文章我將會講到兩件事情
* Pecan Hook
* config file的使用
Pecan Hook
Hook as a middleware, one can use Paste middleware and also combine with hook. The example show as followed. Paste middleware can be used as auth or to check http header for securiy that is indepedently with our core appliction, and hook can be used as database access, that deeply glue with our application.
Firstly, modify *setup_app function in app.py setup_app.
def setup_app(pecan_config=None, extra_hooks=None):
#print 'setup_app pecan_config %s'%pecan_config
app_hooks = [hooks.ConfigHook(),
hooks.SecondHook(),
]
app = pecan.make_app(
pecan_config.app.root,
debug=True,
hooks=app_hooks,
#force_canonical=getattr(pecan_config.app, 'force_canonical', True),
#guess_content_type_from_ext=False
)
#print 'return setup_app'
return appAdding hooks.py in test_project directory.
from pecan import hooks
class ConfigHook(hooks.PecanHook):
@staticmethod
def before(state):
print '---------i am in the hook---------------'
class SecondHook(hooks.PecanHook):
@staticmethod
def before(state):
print '---------i am in the second hook---------------'
@staticmethod
def after(state):
print '---------i am in the second hook in after state---------------'
@staticmethod
def on_route(state):
print '---------i am in the second hook in on-route state---------------'Run the application
root@ubuntu:~/pecan_test/test_project# python api.py
start to run server
start to build paste server
api paste file /root/pecan_test/test_project/api_paste.ini
into app_factory
to get Query Controller
into the root controller
port 8080
* Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)The hook lib contains before, after, on_route function.
The characteritic is where the return query is in the controller.
-----------go away from paste middleware--------
---------i am in the second hook in on-route state---------------
---------i am in the hook---------------
---------i am in the second hook---------------
return query
---------i am in the second hook in after state---------------That means, before() will be executed before into controller, and after() will be executed after into controller. on-route() will be executed before any hook stated. And the Hook function will be execute after Paste middleware.
This example use hook to all application, not in a specific function, of course it supported. But we don't like this kind of writing skill that will be complicate to trace code.
Configuration File
這部分內容可以參考我之前寫的oslo-config的部分
http://gogosatellite.blogspot.tw/2016/05/openstack-oslo-config-tutorial.html
寫任何application都會遇到config file與logging的問題，我們先解決config file的問題。 相信，如果看懂了範例，要將oslo-config放到pecan裡就不是難事了。 此外，python原生也提供config的功能，但，遠遠不及oslo-config的強大，因此對於config的部分，我還是選OpenStack的oslo-config做為使用的基準。
root@ubuntu:~/pecan_test/test_project# ls -l
total 112
-rw-r--r-- 1 root root 363 May 16 23:56 api.conf
-rw-r--r-- 1 root root 585 May 16 23:30 api_paste.ini
-rwxr-xr-x 1 root root 880 May 16 23:39 api.py
-rw-r--r-- 1 root root 5306 May 15 10:17 app_eventlet.py
-rw-r--r-- 1 root root 4645 May 17 00:00 app.py
-rw-r--r-- 1 root root 932 May 14 18:55 app.wsgi
drwxr-xr-x 4 root root 4096 May 14 14:14 build
-rw-r--r-- 1 root root 1020 May 16 18:42 config.py
drwxr-xr-x 2 root root 4096 May 14 14:14 dist
-rw-r--r-- 1 root root 2981 May 16 14:18 healthcheck.py
-rw-r--r-- 1 root root 550 May 16 14:24 hooks.py
-rw-r--r-- 1 root root 0 May 16 23:37 __init__.py
-rw-r--r-- 1 root root 128 May 14 19:14 __init__.pyc
-rw-r--r-- 1 root root 27 May 14 14:04 MANIFEST.in
drwxr-xr-x 4 root root 4096 May 14 14:04 public
-rw-r--r-- 1 root root 14 May 15 11:02 readme
-rw-r--r-- 1 root root 186 May 16 23:35 service.py
-rw-r--r-- 1 root root 96 May 14 14:04 setup.cfg
-rw-r--r-- 1 root root 494 May 14 14:04 setup.py
drwxr-xr-x 6 root root 4096 May 14 19:28 test_project與之前比起來，我們增加了service.py 我們看一下service.py
from oslo_config import cfg
import sys
def prepare_service(argv=None):
if argv is None:
argv = sys.argv
cfg.CONF(argv[1:], project='app', validate_default_values=True)cfg的config file位置，是透過argv來得到的，因此在執行pecan service時，要帶config-file如下:
./api.py --config-file /root/pecan_test/test_project/api.conf此外api.py為入口，config的驅動由此開始。
import app
import service
def main():
service.prepare_service()
print ' start to run server'
app.build_server()
main()其中，service.prepare_service()驅動了config module的啟動。
當然在app.py或者controller的部分，針對config都必須做一些default宣告，並且register到group(section)裡，如下:
from oslo_config import cfg
.
.
CONF = cfg.CONF
OPTS = [
cfg.StrOpt('api_paste_config',
default="api_paste.ini",
help="Configuration file for WSGI definition of API."
),
cfg.IntOpt('api_workers', default=1,
help='Number of workers for Ceilometer API server.'),
cfg.StrOpt('host',
default='0.0.0.0',
help='The listen IP for the ceilometer API server.',
),
cfg.IntOpt('port',
default=8777,
deprecated_name='metering_api_port',
deprecated_group='DEFAULT',
help='The port for the ceilometer API server.',
),
cfg.IntOpt('workers', default=1,
help='Number of workers for Ceilometer API server.'),
cfg.IntOpt('threaded', default=100,
help='Number of threaded for Ceilometer API server.'),
]
API_OPTS = [
cfg.BoolOpt('pecan_debug',
default=False,
help='Toggle Pecan Debug Middleware.'),
]
CONF.register_opts(OPTS)
CONF.register_opts(OPTS, group='api')接下你就可以透過CONF來獲取所要的config資訊。比如
host = CONF.api.host
port = CONF.api.port
workers=CONF.api.workers
threaded=CONF.api.threaded之前範例load_app()部分也可透過config的完成而自動化，程式如下
cfg_file = None
cfg_path = cfg.CONF.api_paste_config
if not os.path.isabs(cfg_path):
cfg_file = CONF.find_file(cfg_path)
elif os.path.exists(cfg_path):
cfg_file = cfg_path
if not cfg_file:
raise cfg.ConfigFilesNotFoundError([cfg.CONF.api_paste_config])
return deploy.loadapp("config:" + cfg_file)接下來就是controller的部分，在不同檔案不同Class下，在我的Blog也介紹過，因此我也不再贅述方法。 於 test_project/controllers/v2/query.py下進行修改，方法與進行app.py時是相同的方式。
QUERY_API_OPTS = [
cfg.IntOpt('threaded', default=100,
help='Number of threaded for Ceilometer API server.'),
]
CONF = cfg.CONF
CONF.register_opts(QUERY_API_OPTS, group='api')接下來在GET query裡我們將config讀出，改變程式如下
@pecan.expose()
def get(self):
print 'show threaded we used %s'%CONF.api.threaded
print 'return query'
logger.error('return query from logger')
#print self.user_id
return "query get"api.conf中的threaded參數可以透過CONF.api.threaded可輕鬆獲得。 因此，透過此方法，你所寫的任何一段程式，在任何檔案，目錄，任何Class都可以獲得。
注意
我們程式部分用的是testproject，但git的部分已經改成testproject_config了，因此，內容與你看到的有些許不同。
Subscribe to:
Posts (Atom)