| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-16 | |||
| 20:00:20 | dansmith | mriedem: it's not now for sure, | |
| 20:00:39 | dansmith | mriedem: and I was thinking of migrations of some sort, but that'll clearly require some special handling anyway | |
| 20:00:39 | mriedem | what apis take multiple instances anyway? | |
| 20:00:53 | dansmith | well, the events api, which is why I said what I did, but that one is already handles | |
| 20:00:55 | dansmith | *handled | |
| 20:01:03 | mriedem | ok | |
| 20:01:34 | mriedem | maybe we'll revive that spec to expose quiesce in the api for multiple instances, which could all be in different cells | |
| 20:01:36 | mriedem | :) | |
| 20:01:51 | mriedem | haha | |
| 20:06:17 | openstackgerrit | Sean Dague proposed openstack/nova master: Verify project_id when quotas are checked https://review.openstack.org/435010 | |
| 20:11:56 | mriedem | sdague: your spec update https://review.openstack.org/#/c/435016/ looks good to me | |
| 20:12:04 | openstackgerrit | Yuming Ma proposed openstack/nova-specs master: Nova BFV by reserving volume https://review.openstack.org/417541 | |
| 20:12:07 | mriedem | it does clarify the logic | |
| 20:13:54 | sdague | it seems to be working on the happy paths, I'll need to put some other real testing in place to test the rest of the cases | |
| 20:17:55 | openstackgerrit | Yuming Ma proposed openstack/nova-specs master: Nova BFV by reserving volume https://review.openstack.org/417541 | |
| 20:39:08 | openstackgerrit | Yuming Ma proposed openstack/nova-specs master: Nova BFV by reserving volume https://review.openstack.org/417541 | |
| 20:40:21 | mriedem | anyone aware of a way to basically squash a series of changes into a single patch using git format-patch? | |
| 20:40:31 | mriedem | without first doing a git rebase to squash them into a single commit | |
| 20:42:16 | mriedem | nvm thanks google http://stackoverflow.com/questions/616556/how-do-you-squash-commits-into-one-patch-with-git-format-patch | |
| 20:46:33 | cdent | edleafe: you the light blue on jay's etherpad? I think we're running into similar mental blocks :( | |
| 20:46:44 | edleafe | dat's me | |
| 20:51:34 | mriedem | nova meeting in 9 minutes | |
| 20:59:42 | openstackgerrit | Maciej Szankin proposed openstack/nova master: Tests: remove .testrepository/times.dbm in tox.ini https://review.openstack.org/435111 | |
| 21:00:14 | mriedem | it's time | |
| 21:35:52 | cfriesen | mriedem: you had proposed that I send in a stable/newton release note for bug 1644248....what's the process for that given that there won't be an equivalent commit in master or stable/ocata? | |
| 21:35:52 | openstack | bug 1644248 in OpenStack Compute (nova) ocata "Nova incorrectly tracks live migration progress" [High,In progress] https://launchpad.net/bugs/1644248 - Assigned to Matt Riedemann (mriedem) | |
| 21:36:44 | mriedem | cfriesen: just commit a release note to stable/newton with the known issue listed | |
| 21:37:39 | cfriesen | mriedem: okay, can do | |
| 21:42:11 | openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: spawn/destroy https://review.openstack.org/391288 | |
| 21:59:10 | openstackgerrit | melanie witt proposed openstack/nova master: Count resources to check quota for cells https://review.openstack.org/416521 | |
| 22:00:31 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.42 - Fix tag attribute disappearing https://review.openstack.org/429512 | |
| 22:02:17 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.43: List/Show all server migration types https://review.openstack.org/430839 | |
| 22:02:29 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.43 - Enable cold migration with target host https://review.openstack.org/406707 | |
| 22:02:43 | openstackgerrit | Dan Smith proposed openstack/nova master: Make servers API use cell-targeted context https://review.openstack.org/433362 | |
| 22:02:49 | dansmith | melwitt: surprisingly little drama with unit and functional ^ | |
| 22:02:56 | dansmith | we'll see how devstack likes it | |
| 22:04:41 | melwitt | dansmith: yeah, that's almost no change to the unit and functional. huh. | |
| 22:05:05 | openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: power_on/off and reboot https://review.openstack.org/427380 | |
| 22:05:19 | dansmith | well, it kinda makes sense since I'm just making the set call a no-op for the simple cell fixture | |
| 22:05:22 | dansmith | although I'm surprised a bunch of functional tests didn't fail | |
| 22:05:26 | dansmith | but we'll see | |
| 22:06:19 | melwitt | yeah | |
| 22:06:20 | mriedem | bauzas: got it for you https://review.openstack.org/#/c/433855/ | |
| 22:08:43 | bauzas | mriedem: heh | |
| 22:09:05 | bauzas | anyway, I promised a comment | |
| 22:12:02 | openstackgerrit | Sujitha proposed openstack/nova master: Change tags to default field in Instance object. https://review.openstack.org/415298 | |
| 22:12:55 | cfriesen | dansmith: regarding https://review.openstack.org/#/c/265282/ if we remove the "name" and "id" from the flavor details, do we just display all of the flavor info when doing "nova list --fields flavor"? Currently it shows just the flavor ID. | |
| 22:21:44 | zzfoo | Hi, I'm trying to get live migration working. I have shared store setup and working with NFS though when i do a live migrate I get the following error in the nova-compute logs: error: | |
| 22:21:44 | zzfoo | Unable to acquire lock on `/opt/stack/data/nova/instances/locks/nova-fd2836e4eef36d5483a37a5000a42a72fb95ea7c` due to [Errno 37] No locks available | |
| 22:22:09 | cfriesen | does anyone know why I'm not getting an auto-generated Change-Id when creating a python-novaclient commit? Is this expected or did I mess something up? | |
| 22:22:25 | zzfoo | on the shared storage though that file has been created. | |
| 22:22:49 | melwitt | cfriesen: not expected. did you 'git review -s' first after you cloned python-novaclient? | |
| 22:23:07 | melwitt | you have to do it anytime you clone a new rep, afaik | |
| 22:23:10 | melwitt | *repo | |
| 22:23:19 | cfriesen | melwitt: it's entirely possible that I didn't | |
| 22:24:13 | openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: localdisk https://review.openstack.org/409401 | |
| 22:24:51 | melwitt | cfriesen: you can try 'git review -s' in the repo dir, then 'git commit --amend' and then you should get a change-id | |
| 22:25:12 | cfriesen | melwitt: that seems to have done it. thanks | |
| 22:32:28 | EmilienM | mriedem: I took the freedom to write some thoughts on https://etherpad.openstack.org/p/nova-ocata-retrospective - I'll probably have more later | |
| 22:32:56 | openstackgerrit | Chris Friesen proposed openstack/python-novaclient master: Add support for showing flavor details (v2.43) https://review.openstack.org/435141 | |
| 22:33:50 | mriedem | EmilienM: ok, i expected it :) | |
| 22:40:10 | mriedem | hmm, what runs qemu-img info in a periodic? | |
| 22:40:16 | mriedem | get_available_resources? | |
| 22:42:15 | cfriesen | seems likely | |
| 22:42:29 | mriedem | so, i've seen this a few times lately, | |
| 22:42:39 | mriedem | the live migration job fails because we timeout waiting for an instance to spawn | |
| 22:42:43 | mriedem | and we get here http://logs.openstack.org/50/435050/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/49adcde/logs/subnode-2/screen-n-cpu.txt.gz#_2017-02-16_22_22_11_143 | |
| 22:42:51 | mriedem | 2017-02-16 22:22:11.143 14954 DEBUG nova.virt.libvirt.driver [req-025bf4d2-e5ba-4236-a334-f9eb98105ada tempest-LiveBlockMigrationTestJSON-793129345 tempest-LiveBlockMigrationTestJSON-793129345] [instance: 0ee93807-d206-4ddf-878c-efd1dd2eab3c] Instance is running spawn /opt/stack/new/nova/nova/virt/libvirt/driver.py:2689 | |
| 22:43:01 | mriedem | which calls spawn() on the libvirt driver | |
| 22:43:11 | mriedem | and after that, it's radio silence on that instance and request except for periodic tasks | |
| 22:43:26 | mriedem | i have a feeling we have some kind of deadlock or something in the libvirt driver's spawn flow | |
| 22:43:37 | mriedem | i've seen it since newton | |
| 22:47:37 | cfriesen | so you're thinking self._domain.createWithFlags() is triggering something wonky? | |
| 22:48:36 | cfriesen | or is there another async call before that? | |
| 22:50:57 | openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: console https://review.openstack.org/409402 | |
| 22:51:30 | mriedem | cfriesen: i don't know what it is | |
| 22:51:41 | mriedem | last time i looked into this i was going to add some debug logging in the image create code | |
| 22:51:50 | mriedem | because that's gone through a lot of churn since newton | |
| 22:53:46 | cfriesen | mriedem: we're getting back to the debug log though, which means that we've done the image create, no? | |
| 22:54:58 | mriedem | this is also weird http://logs.openstack.org/50/435050/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/49adcde/logs/subnode-2/screen-n-cpu.txt.gz#_2017-02-16_22_22_06_286 | |
| 22:55:04 | mriedem | Instance network_info: |[]| | |
| 22:57:21 | mriedem | so i wonder if, | |
| 22:57:33 | mriedem | the compute manager is just waiting for the async network_info stuff to get done | |
| 22:57:54 | mriedem | but it doesn't complete because it's empty - but i don't think that's how it works anyway, we allow booting a server with no networkin | |
| 22:58:02 | mriedem | and test it in tempest with the 2.37 microversion | |
| 22:58:45 | cfriesen | whatever it is, it's not blocking the self._create_domain_and_network() call because we're seeing the log right after it. So it has to be something async that we're not waiting for. | |
| 23:02:38 | mriedem | cfriesen: yeah right after that we see timer = loopingcall.FixedIntervalLoopingCall(_wait_for_boot) | |
| 23:02:51 | mriedem | which is waiting for the power state to be RUNNING | |
| 23:03:01 | mriedem | i guess we just never finish there | |
| 23:03:09 | mriedem | because i never see "Instance spawned successfully." | |
| 23:03:42 | mriedem | even though i see http://logs.openstack.org/50/435050/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/49adcde/logs/subnode-2/screen-n-cpu.txt.gz#_2017-02-16_22_22_11_214 | |
| 23:03:48 | mriedem | [instance: 0ee93807-d206-4ddf-878c-efd1dd2eab3c] VM Started (Lifecycle Event) | |
| 23:04:06 | mriedem | and then right after that [instance: 0ee93807-d206-4ddf-878c-efd1dd2eab3c] VM Paused (Lifecycle Event) | |
| 23:04:44 | mriedem | Synchronizing instance power state after lifecycle event "Paused"; current vm_state: building, current task_state: spawning, current DB power_state: 0, VM power_state: 3 | |
| 23:04:50 | mriedem | 3 = paused | |
| 23:05:53 | mriedem | i don't see "Instance spawned successfully." anywhere in http://logs.openstack.org/50/435050/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/49adcde/logs/subnode-2/screen-n-cpu.txt.gz | |
| 23:06:51 | mriedem | so the instance goes from started to paused, | |
| 23:06:56 | mriedem | and then is never unpaused | |
| 23:07:06 | mriedem | which is why the spawn() method never considers it done i guess | |
| 23:08:07 | mriedem | http://logs.openstack.org/50/435050/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/49adcde/logs/subnode-2/libvirt/qemu/instance-00000001.txt.gz | |