| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-11 | |||
| 20:35:24 | kevi9132 | I'm sure I'm missing something so any guidance would be greatly appreciated. | |
| 20:38:40 | mriedem | kevi9132: do you have a paste bin of the error in the compute logs? | |
| 20:38:57 | mriedem | assuming builds are failing in the scheduler with NoValidHost? | |
| 20:39:57 | mriedem | anything hitting placement APIs should be decorated with this https://github.com/openstack/nova/blob/stable/newton/nova/scheduler/client/report.py#L49 so that we handle failures and just log warnings | |
| 20:40:52 | kevi9132 | Let me try to get the failure generated again since I smoked my VM. IIRC, the failure was NoValidHost because no hosts matched the RAM filter | |
| 20:44:03 | kevi9132 | I believe the decorator is being used. The exception is caused because an auth plugin is not found trying to create/resolve resources which results in no resource available for an instance build. | |
| 21:18:47 | kevi9132 | So, after booting the VM, I see that each row in the compute_nodes table has free ram and disk. | |
| 21:19:54 | kevi9132 | I'm guessing that the exception I was experiencing is expected and is now a red herring to what my instance build issue really is. I'm building an instance now and will capture the failure in the nova logs and post it. | |
| 21:25:22 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/502700 | |
| 21:27:47 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/502708 | |
| 21:28:59 | mriedem1 | dansmith: so does approving of this ironic flavor migration offline data migration thing mean that we're setting a precedent that now all data migrations will need both online and offline modes supported? | |
| 21:29:10 | mriedem | i'm guessing yes | |
| 21:29:30 | mriedem | unless there is a caveat about when we do a hard drop of something that relies on the data being migrated in the next release | |
| 21:30:16 | openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/502745 | |
| 21:34:46 | dansmith | mriedem: we've always had that for the online migrations because of the online_data_migrations command, IMHO | |
| 21:35:02 | dansmith | and the skip level people are definitely asking for always having an offline option for that use case | |
| 21:35:07 | dansmith | which I think is very reasonable | |
| 21:35:28 | dansmith | mriedem: the offline mode is specifically so they can run a thing before a release drops a thing | |
| 21:35:41 | mriedem | "we've always had that for the online migrations because of the online_data_migrations command, IMHO" | |
| 21:35:49 | mriedem | we've always had online data migrations since online_data_migrations, i agree | |
| 21:36:07 | dansmith | mriedem: so we can drop those migrations, we just need to have them in a release so people can run them without services starting | |
| 21:36:37 | mriedem | so in the before times, before online data migrations, we'd just do data migrations in the sqla-migrate scripts right? | |
| 21:36:42 | dansmith | yeah | |
| 21:36:58 | dansmith | and online_data_migrations was somewhat weirdly named, in that it was the way to run the online migrations manually | |
| 21:37:01 | mriedem | and we do'nt want to do these there since lots of people don't want the migration happening during downtime | |
| 21:37:08 | dansmith | yah | |
| 21:37:42 | mriedem | i just wanted to make sure we realize what we're signing up for | |
| 21:38:11 | dansmith | I guess I don't see it as a change from what we've been doing | |
| 21:38:33 | mriedem | we haven't been doing offline migrations | |
| 21:38:39 | melwitt | yeah, this one is the only one we haven't provided a manual run for, right? | |
| 21:38:56 | melwitt | that is, provided a nova-manage way to run it (via online_data_migrations) | |
| 21:39:26 | dansmith | we had one previously for pci stuff that was like the very first one | |
| 21:39:51 | mriedem | we have 3 other online data migrations added in pike, why wouldn't we also be providing offline options for those? | |
| 21:39:55 | dansmith | mriedem: we /have/ been providing them by having this command | |
| 21:40:11 | melwitt | yeah, that's what nova-manage online_data_migrations is | |
| 21:40:21 | dansmith | yeah | |
| 21:40:22 | dansmith | exactly | |
| 21:40:36 | melwitt | what we haven't had the past few times is actual active online data migrations (sorry that's confusing) | |
| 21:40:37 | dansmith | it's the offline way to run online migrations ... always have been | |
| 21:41:03 | dansmith | yeah, some of the ones we've had recently didn't actually do online background stuff and only happened as part of this command, but.. it's the same thing | |
| 21:41:10 | melwitt | right | |
| 21:41:24 | mriedem | huh? | |
| 21:41:38 | mriedem | so we've always required that everything is down when we run online_data_migrations? | |
| 21:41:51 | melwitt | like, the past few data migrations we did, were not done actively in the background automatically while nova runs | |
| 21:41:58 | mriedem | https://docs.openstack.org/nova/latest/user/upgrade.html#rolling-upgrade-process | |
| 21:42:08 | mriedem | "Start all services on the new code, with [upgrade_levels]compute=auto in nova.conf. It is safest to start nova-conductor first and nova-api last." | |
| 21:42:29 | melwitt | no, we don't require that things are down. it's just that we aren't doing it automatically in the background, we provided only the manual nova-manage command to take care of it | |
| 21:42:31 | mriedem | (later step): "This process can put significant extra write load on the database. Complete all online data migrations using: nova-manage db online_data_migrations --max-count <number>. " | |
| 21:42:47 | dansmith | I think we're stuck on terminology here | |
| 21:43:07 | dansmith | mriedem: where are you right now? we're about done here and this might be easier in person | |
| 21:43:26 | mriedem | i'm cranky now and hiding | |
| 21:43:29 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/502700 | |
| 21:43:32 | dansmith | alright | |
| 21:46:26 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: doc: note that custom resources are not fully supported https://review.openstack.org/501252 | |
| 21:46:38 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/502708 | |
| 21:47:24 | mriedem | i'm on a couch in a corner by the EMC business area | |
| 21:49:09 | openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/502745 | |
| 21:54:16 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/pike: doc: note that custom resources are not fully supported https://review.openstack.org/500521 | |
| 22:05:16 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Refactor ServerMovingTests for non-move tests https://review.openstack.org/498596 | |
| 22:38:28 | rybridges | Hello. I had a question or 2 about vendordata in the Ocata release | |
| 22:39:39 | rybridges | Are we still able to write our vendordata logic stuff into a driver? I am noticing this line in the ocata code base -> https://github.com/openstack/nova/blob/stable/ocata/nova/api/metadata/base.py#L745 Is vendordata not doable through a driver anymore? | |
| 23:07:20 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Merge extended availability zone controller https://review.openstack.org/502859 | |
| #openstack-nova - 2017-09-12 | |||
| 00:06:50 | mwynne | Hey guys. I'm looking at the rabbit queues (rabbitmqctl list_queues | awk '$2 != 0{print}'), and noticed that the conductor queue is always increasing. Is this normal? Can I debug this somehow? | |
| 00:12:37 | mwynne | netstat -tunlp | |
| 00:12:47 | mwynne | (whoops) | |
| 04:09:16 | openstackgerrit | Merged openstack/nova master: Refactor ServerMovingTests for non-move tests https://review.openstack.org/498596 | |
| 04:21:31 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/502700 | |
| 07:23:04 | rabel | can someone review https://review.openstack.org/#/c/494169/ ? i think except from minor changes it is ready for weeks. | |
| 08:02:11 | openstackgerrit | Christian Berendt proposed openstack/nova master: Choose VMware datastore by the least provisioned space https://review.openstack.org/402632 | |
| 08:51:25 | rabel | https://review.openstack.org/#/c/402632/10 seems to be ready for final review for months. could someone have a look at it? | |
| 10:07:45 | mewald | Hey guys, is there a chance the block_device_mapping table can be truncated / cleared without any problems? Nova should be able to get this info from Cinder via API, right? | |
| 10:28:38 | mewald | ok forget it, it was bullshit | |
| 10:34:01 | mewald | I have a wrong / incomplete connection_info column in block_device_mapping table. New volumes are added with different connection_info JSON objects than: for example they include the secret_uuid. The lack of this uuid is currently breaking my set up . Any advice on how to fix this? | |
| 11:58:45 | bhagyashris | Hi all, Need guidance of How to write the upgrade script? | |
| 13:29:18 | boolman | I'm having problems adding iolimits on my vms. I tried adding quota on flavor,image,volume but it won't add iotune to the virsh DOM. output from image/volume/flavor show : http://ix.io/zMX | |
| 13:52:53 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova-specs master: Intel Fortville Dynamic Device Personalization (DDP) https://review.openstack.org/503001 | |
| 14:06:15 | openstackgerrit | Merged openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/502745 | |
| 14:07:57 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/502700 | |
| 14:09:05 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova-specs master: Intel Fortville Dynamic Device Personalization (DDP) https://review.openstack.org/503001 | |
| 14:11:07 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Migration from ``ip`` commands to ``pyroute2`` https://review.openstack.org/484386 | |
| 14:18:58 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transform instance.resize.error notifications https://review.openstack.org/417882 | |
| 14:20:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: fake_notifier: Refactor wait_for_versioned_notification https://review.openstack.org/489637 | |
| 14:38:44 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova-specs master: Intel Fortville Dynamic Device Personalization (DDP) https://review.openstack.org/503001 | |
| 15:00:53 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(1/2) https://review.openstack.org/408955 | |
| 15:06:12 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(2/2) https://review.openstack.org/408964 | |
| 15:06:29 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Add parameters in cold migrate action https://review.openstack.org/410042 | |
| 15:14:14 | mriedem | bauzas: are you ok with this? https://review.openstack.org/#/c/499399/ - you said you were happy after the reno got updated | |
| 15:14:24 | mriedem | i'd like to move this series along to get it backported to stable/pike | |
| 15:17:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove dest node allocation if evacuate MoveClaim fails https://review.openstack.org/499878 | |
| 15:17:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add a test to make sure failed evacuate cleans up dest allocation https://review.openstack.org/499877 | |
| 15:17:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Pass migration from API to conductor for evacuate https://review.openstack.org/500176 | |
| 15:17:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Create allocations against forced dest host during evacuate https://review.openstack.org/499399 | |
| 15:17:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for evacuate claim failure https://review.openstack.org/499874 | |
| 15:17:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Modernize set_vm_state_and_notify https://review.openstack.org/499799 | |
| 15:31:19 | mriedem | Kevin_Zheng: yes if you go overquota after the build_request is created when we recheck in conductor, the instance will be put into ERROR state https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1084 | |
| 15:31:20 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: factor out compute service start in ServerMovingTest https://review.openstack.org/503037 | |
| 15:31:20 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Moving more utils to ServerResourceAllocationTestBase https://review.openstack.org/499539 | |
| 15:31:36 | mriedem | Kevin_Zheng: that's because we can't just delete the instance at that point in conductor | |
| 15:31:53 | mriedem | because the use could be listing the instance via the BuildRequest before it goes to ERROR state | |
| 15:31:56 | mriedem | *user | |