file injection代码
file injection原理来讲是比较简单的,在nova boot命令中,有参数--file,是将文件inject到image中
nova boot --flavor 2 --image d96b0e41-8264-41de-8dbb-6b31ce9bfbfc --key-name openstack --security-groups default --file /home/ubuntu/bootfromvolume1.xml=/home/cliu8/images/bootfromvolume.xml testinject19
这里容易混淆的是还有一个参数是user_data,这不是file injection,而是cloud-init使用这里面的东西对系统进行初始化。
在虚拟机启动的时候,会调用到/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py中的spawn函数
def spawn(self, context, instance, image_meta, injected_files, admin_password, network_info=None, block_device_info=None): disk_info = blockinfo.get_disk_info(CONF.libvirt.virt_type, instance, block_device_info, image_meta) self._create_image(context, instance, disk_info['mapping'], network_info=network_info, block_device_info=block_device_info, files=injected_files, admin_pass=admin_password) xml = self.to_xml(context, instance, network_info, disk_info, image_meta, block_device_info=block_device_info, write_to_disk=True)
self._create_domain_and_network(context, xml, instance, network_info, block_device_info) LOG.debug(_("Instance is running"), instance=instance)
其中一步是create image,在_create_image函数中,有下面的逻辑
elif inject_files and CONF.libvirt.inject_partition != -2: if booted_from_volume: LOG.warn(_('File injection into a boot from volume ' 'instance is not supported'), instance=instance) self._inject_data( instance, network_info, admin_pass, files, suffix)
当有文件要inject的时候,并且参数inject_partition不设为-2,则执行file injection过程。
从这里我们可以看出,在参数中应该添加
# cat /etc/nova/nova-compute.conf [DEFAULT]compute_driver=libvirt.LibvirtDriver[libvirt]virt_type=kvminject_partition=-1
在代码中,我们可以看到这个参数是这样解释的:
cfg.IntOpt('inject_partition', default=-2, help='The partition to inject to : ' '-2 => disable, -1 => inspect (libguestfs only), ' '0 => not partitioned, >0 => partition number', deprecated_name='libvirt_inject_partition', deprecated_group='DEFAULT'),
在最新的代码中,我们往往选择用ligguestfs来进行文件注入,因而设为-1
在_inject_data函数中,
disk.inject_data(injection_path, key, net, metadata, admin_pass, files, partition=target_partition, use_cow=CONF.use_cow_images, mandatory=('files',))
(Pdb) p injection_path'/var/lib/nova/instances/fedc59a9-6ff5-4c23-9595-d877a6796c84/disk'
(Pdb) p target_partition-1
这里key, net, metadata, admin_pass我们都不注入,所以都为空,只有files有值
这里就调用到/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py的
def inject_data(image, key=None, net=None, metadata=None, admin_password=None, files=None, partition=None, use_cow=False, mandatory=()):
在这个函数中首先生成一个对象来操作这个image
fs = vfs.VFS.instance_for_image(image, fmt, partition)
如果能够找到guestfs,则fs为nova.virt.disk.vfs.guestfs.VFSGuestFS
接着调用来初始化环境,该run起来的guestfs的applicance要运行起来,该mount的image要mount上去
fs.setup()
最后进行注入操作
return inject_data_into_fs(fs, key, net, metadata, admin_password, files, mandatory)
在/usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/guestfs.py的setup函数中,
并不是调用guestfish命令,而是直接通过python调用guestfsd的API,可以参考http://libguestfs.org/guestfs.3.html
先要有个handle
self.handle = tpool.Proxy(guestfs.GuestFS(close_on_exit=False))
然后将image作为一个drive添加到appliance
self.handle.add_drive_opts(self.imgfile, format=self.imgfmt)
运行appliance
self.handle.launch()
self.setup_os()
self.handle.aug_init("/", 0)
setup_os的实现很简单,由于我们的参数是-1,则调用setup_os_inspect
def setup_os(self): if self.partition == -1: self.setup_os_inspect() else: self.setup_os_static()
在函数setup_os_inspect中
先看有多少个root disk
roots = self.handle.inspect_os()
(Pdb) p roots['/dev/sda1']
发现只有一个然后调用
self.setup_os_root(roots[0])
在setup_os_root中,先是得到mount point
mounts = self.handle.inspect_get_mountpoints(root)
(Pdb) p mounts[('/', '/dev/sda1')]
最后self.handle.mount_options("", mount[1], mount[0])
(Pdb) p mount[1]'/dev/sda1'(Pdb) p mount[0]'/'
将/dev/sda1挂载到/下面
至此appliance起来了,image也挂载了, fs.setup()成功,接下来是调用inject_data_into_fs
inject_data_into_fs对不同的注入对象调用不同的函数,对于文件注入调用def _inject_files_into_fs(files, fs):
这里面是一个循环,每个文件都调用def _inject_file_into_fs(fs, path, contents, append=False):
这里面调用了fs.replace_file(path, contents)
其实是写一个文件到一个path
def replace_file(self, path, content): LOG.debug(_("Replace file path=%s"), path) path = self._canonicalize_path(path) self.handle.write(path, content)
(Pdb) p pathu'/home/ubuntu/bootfromvolume1.xml'(Pdb) p content"
至此文件注入完毕。
file injection调通过程
上面的过程,理论上很简单,但是真正调通,还是经历了复杂的过程
我的环境是Ubuntu 14.04, apt-get安装的Icehouse
首先是这个错误
2014-07-09 22:50:12.359 7885 WARNING nova.virt.disk.vfs.guestfs [req-b3655a75-1607-4c90-b7cb-cc81e543e769 5df0e89888364c0b80d47a7a426a9a67 c24c59846a7f44538d958e7548cc74a3] Failed to close augeas aug_close: call launch before using this function(in guestfish, don't forget to use the 'run' command)2014-07-09 22:50:12.366 7885 ERROR nova.virt.libvirt.driver [req-b3655a75-1607-4c90-b7cb-cc81e543e769 5df0e89888364c0b80d47a7a426a9a67 c24c59846a7f44538d958e7548cc74a3] [instance: 7bf0f046-69a7-4c5f-990b-9cbfaf3b0b55] Error injecting data into imaged96b0e41-8264-41de-8dbb-6b31ce9bfbfc (Error mounting /var/lib/nova/instances/7bf0f046-69a7-4c5f-990b-9cbfaf3b0b55/disk with libguestfs (cannot find any suitable libguestfs supermin, fixed or old-style appliance on LIBGUESTFS_PATH (search path: /usr/lib/guestfs)))
参考http://libguestfs.org/guestfs-faq.1.html
libguestfs: error: cannot find any suitable libguestfs supermin, fixed or old-style appliance on LIBGUESTFS_PATH
febootstrap-supermin-helper: ext2: parent directory not found
supermin-helper: ext2: parent directory not found
[This issue is fixed permanently in libguestfs ≥ 1.26.]
If you see any of these errors on Debian/Ubuntu, you need to run the following command:
sudo update-guestfs-appliance
我运行了update-guestfs-appliance,可是还是不管用,那就升级吧
ubuntu 14.04的repository里面的guestfs是1.24,于是我下载了最新的1.26,然后从代码安装
wget http://libguestfs.org/download/1.26-stable/libguestfs-1.26.0.tar.gz
tar xvzf libguestfs-1.26.0.tar.gz
cd libguestfs-1.26.0
apt-get install build-essential
首先用apt-get purge将原来安装的所有的guestfs的package都删除
为了能够编译成功,安装了下面的包
apt-get install libncurses5 libncurses5-dev ocaml ocaml-findlib-wizard ocaml-findlib libfindlib-ocaml libfindlib-ocaml-dev pk e2fslibs e2fslibs-dev gperf flex bison libpcre3 libpcre3-dev libaugeas-dev ncurses-base libcurses-ocaml ncurses-bin ncurses-dev erlang-dev
当然可能还不局限于这些包,在configure, make, make install的过程中,遇到没有的包,就apt-cache search,然后install
在configure libguestfs之前,supermin也需要升级
wget http://libguestfs.org/download/supermin/supermin-5.1.8.tar.gz
tar xvzf supermin-5.1.8.tar.gz
cd supermin-5.1.8
./configure
make
make install
上面的有些包是为安装supermin准备的。
在configure libguestfs之前,要export LIBS="-lncurses",要不他会找不到tgetent和tgetnum
而且configure的是,会对qemu的版本做检查
在configure.ac中
772 # AC_MSG_CHECKING([for $QEMU version >= 1])773 # if $QEMU -version | grep -sq 'version @<:@1-@:>@'; then774 # AC_MSG_RESULT([yes])775 # else776 # AC_MSG_RESULT([no])777 # AC_MSG_FAILURE([$QEMU version must be >= 1.0.])778 # fi
这里要求qemu的版本一定是1.x
很不幸
# qemu-system-x86_64 --versionQEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1), Copyright (c) 2003-2008 Fabrice Bellard
我的是2.x,反而报错误说我的版本老了,没办法,只好注释掉
修改了configure.ac,于是需要安装apt-get install autoconf
运行autoconf重新生成configure文件
下面configure libguestfs,需要指定qemu
./configure --with-qemu="qemu-system-x86_64"
中间有任何的错误都必须google,然后让他过,不能忽略,因为libguestfs的工具和包太多,说不定哪个不起作用,都会影响文件注入
make
make install
中间有.so, .a冲突,要把原来的弄掉
cd /
find . -name "dllmlguestfs.so"
cd /usr/lib/ocaml/stublibs/
mv dllmlguestfs.so dllgraphics.so.back
重新make install,直到正常成功结束。
安装成功了,试一把
virt-ls -a ubuntutest.img /home/openstack
似乎成功
guestfish -a ubuntutest.img
里面运行run
似乎也成功
就基本可以了
接下来创建虚拟机,带文件注入
可还是报这个错误
2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1355, in _build_instance2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] filter_properties, bdms, legacy_bdm_in_spec)2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1401, in _reschedule_or_error2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] self._log_original_error(exc_info, instance_uuid)2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] six.reraise(self.type_, self.value, self.tb)2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1396, in _reschedule_or_error2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] bdms, requested_networks)2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2125, in _shutdown_instance2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] requested_networks)2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] six.reraise(self.type_, self.value, self.tb)2014-07-10 08:11:53.226 2550 TRACE nova.compute.utils [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2115, in _shutdown_instance2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] instance, network_info, admin_pass, files, suffix)2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2539, in _inject_data2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] instance=instance)2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] six.reraise(self.type_, self.value, self.tb)2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] File /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2533, in _inject_data2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] mandatory=('files',))2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] File "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 353, in inject_data2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] fs.setup()2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] File /usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/guestfs.py", line 132, in setup2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] {'imgfile': self.imgfile, 'e': e})2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] NovaException: Error mounting /var/lib/nova/instances/e92f42dd-4164-4dff-a6a7-d42bd04be672/disk with libguestfs (/usr/local/bin/supermin exited with error status 1.2014-07-10 08:11:53.152 2550 TRACE nova.compute.manager [instance: e92f42dd-4164-4dff-a6a7-d42bd04be672] To see full error messages you may need to enable debugging.
为什么,不明觉厉
用pdb debug一把(在root下)
python -m pdb nova-pdb-compute.py --config-file=/etc/nova/nova.conf --config-file=/etc/nova/nova-compute.conf
# cat nova-pdb-compute.py#! /usr/bin/python# PBR Generated from u'console_scripts'
import sysimport pdb
from nova.cmd.compute import main
if __name__ == "__main__": sys.exit(main())
就是从/usr/bin/nova-compute拷贝过来,前面加上import pdb
单步运行,竟然通过了,想不出什么区别
突然看到
# ps aux | grep nova-computeroot 22745 0.0 0.0 10464 928 pts/3 S+ 19:37 0:00 grep --color=auto nova-computenova 29488 1.0 0.1 1761032 75100 ? Ssl 17:25 1:20 /usr/bin/python /usr/bin/nova-compute --config-file=/etc/nova/nova.conf --config-file=/etc/nova/nova-compute.conf
service nova-compute start启动的进程不是以root运行的,而是以nova用户运行的
于是service nova-compute stop
然后在root下运行
/usr/bin/python /usr/bin/nova-compute --config-file=/etc/nova/nova.conf --config-file=/etc/nova/nova-compute.conf > nova-compute.log 2>&1 &
果然成功了。
可见普通用户运行guestfish还是有问题的,上面的所有操作都是在root下,于是切换到普通用户
cliu8:~/images$ guestfish -a ubuntutest.img
Welcome to guestfish, the guest filesystem shell forediting virtual machine filesystems and disk images.
Type: 'help' for help on commands 'man' to read the manual 'quit' to quit the shell
>
哈哈,在普通用户下,果然run不起appliance来
usermod -G kvm -a novausermod -G kvm -a cliu8usermod -G root -a cliu8 usermod -G root -a nova
将普通用户和nova都添加到kvm, root的group
打开debug,再运行guestfish
cliu8:~/images$ export LIBGUESTFS_DEBUG=1cliu8:~/images$ export LIBGUESTFS_TRACE=1cliu8:~/images$ guestfish -a ubuntutest.img libguestfs: trace: set_verbose truelibguestfs: trace: set_verbose = 0libguestfs: create: flags = 0, handle = 0x196ba50, program = guestfishlibguestfs: trace: set_pgroup truelibguestfs: trace: set_pgroup = 0libguestfs: trace: add_drive "ubuntutest.img"libguestfs: trace: add_drive = 0
Welcome to guestfish, the guest filesystem shell forediting virtual machine filesystems and disk images.
Type: 'help' for help on commands 'man' to read the manual 'quit' to quit the shell
>
竟然还是不行
最后终于发现
http://libguestfs.org/guestfs-faq.1.html
DOWNLOADING, INSTALLING, COMPILING LIBGUESTFS
Debian and Ubuntu After installing libguestfs you need to do:
sudo update-guestfs-appliance
On Ubuntu only:
sudo chmod 0644 /boot/vmlinuz*
You may need to add yourself to the kvm group:
sudo usermod -a -G kvm yourlogin
真是不看文档,吃亏在眼前啊
赶紧
chmod 0644 /boot/vmlinuz*
然后运行guestfish -a ubuntutest.img,里面run,哈哈成功了
于是可以再nova用户下运行nova-compute了
service nova-compute start
再order一个机器,带文件注入的
成了
# ip netns exec qrouter-26a45e0e-a58a-443b-a972-d62c0c5a1323 ssh -i openstack.pem ubuntu@192.168.0.33The authenticity of host '192.168.0.33 (192.168.0.33)' can't be established.ECDSA key fingerprint is 20:f6:58:a9:c0:ab:54:7e:5f:3b:1f:3c:75:c6:36:26.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added '192.168.0.33' (ECDSA) to the list of known hosts.Welcome to Ubuntu 12.04.4 LTS (GNU/Linux 3.2.0-64-virtual x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Thu Jul 10 11:48:11 UTC 2014
System load: 0.0 Processes: 59 Usage of /: 4.0% of 19.68GB Users logged in: 0 Memory usage: 3% IP address for eth0: 192.168.0.33 Swap usage: 0%
Graph this data and manage this system at:https://landscape.canonical.com/
Get cloud support with Ubuntu Advantage Cloud Guest:http://www.ubuntu.com/business/services/cloud
0 packages can be updated.0 updates are security updates.
The programs included with the Ubuntu system are free software;the exact distribution terms for each program are described in theindividual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted byapplicable law.
To run a command as administrator (user "root"), use "sudo
ubuntu@testinject15:~$ lsbootfromvolume1.xml
发表评论