| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-27 | |||
| 14:08:00 | mriedem | tssurya: nova-manage uses nova.conf if no other --config-file is specified | |
| 14:08:02 | mriedem | on the command line | |
| 14:08:31 | mriedem | so if you want to do things to cell1, you would do something like: nova-manage --config-file /etc/nova/nova_cell1.conf db archive_deleted_instances --verbose --until-complete | |
| 14:08:37 | mriedem | for archiving deleted instances in cell1 | |
| 14:08:52 | stephenfin | efried: You didn't start work converting this to a seqdiag, did you? https://photos.google.com/share/AF1QipNpWVQKU8GK4_9wxVbiRJUqJnMzqPcBh6DvjVyBPIjjmi6ZU8r9TleQNo6pV1t9SA?key=NUl3OUFGYkRFTE8tMHhSX0lfc0Y1eEdoeHo4SUhn | |
| 14:08:55 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP(5): PowerVM driver: ovs vif https://review.openstack.org/422512 | |
| 14:09:01 | stephenfin | I think mriedem asked for it? | |
| 14:09:04 | tssurya | mriedem, dansmith : okay thanks that makes sense | |
| 14:09:13 | dansmith | tssurya: yeah look at the docstring on that method, it says it assumes the config points at the right database | |
| 14:09:30 | mriedem | tssurya: related https://bugs.launchpad.net/nova/+bug/1719487 | |
| 14:09:32 | openstack | Launchpad bug 1719487 in OpenStack Compute (nova) "nova-manage db archive_deleted_rows is not multi-cell aware" [Wishlist,Triaged] - Assigned to Zhenyu Zheng (zhengzhenyu) | |
| 14:09:39 | dansmith | tssurya: you could make it smarter :) | |
| 14:09:40 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146 | |
| 14:09:43 | mriedem | stephenfin: that's already done | |
| 14:09:59 | mriedem | stephenfin: https://docs.openstack.org/nova/latest/reference/live-migration.html | |
| 14:10:01 | efried | stephenfin Yes, merged. | |
| 14:10:03 | mriedem | there is one thing missing | |
| 14:10:07 | mriedem | i noticed yesterday | |
| 14:10:31 | mriedem | on the failure path, it doesn't have a box saying that the source node is running the _rollback_live_migration method | |
| 14:10:42 | mriedem | which does some stuff on the source node, and calls the dest node to cleanup | |
| 14:10:49 | tssurya | dansmith , mriedem : ok, will have a look at it :) | |
| 14:11:17 | openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements https://review.openstack.org/493187 | |
| 14:11:26 | efried | mriedem That would be like a box to the left of that final 'call' arrow at the bottom? | |
| 14:11:54 | mriedem | efried: i think more like how the post_live_migration box is in the success path on the source node | |
| 14:14:40 | efried | sdague cdent Care to render an opinion/ruling on jichenjc's concerns here: https://review.openstack.org/#/c/488137/21/nova/conf/utils.py@85 ? | |
| 14:17:01 | cdent | efried: remind me when I’m back in about 20 mins | |
| 14:17:08 | efried | ack, thx | |
| 14:17:55 | openstackgerrit | Eric Fried proposed openstack/nova master: _rollback_live_migration in live-migration seqdiag https://review.openstack.org/507871 | |
| 14:18:23 | efried | mriedem ^ | |
| 14:19:26 | mriedem | thanks | |
| 14:19:56 | mriedem | claudiub|3: hyperv ci seems to have gone crazy | |
| 14:19:57 | mriedem | http://cloudbase-ci.com//nova/507871/1/console.log.gz | |
| 14:20:08 | mriedem | not sure if that's a zuulv3 side effect or what | |
| 14:22:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: xenapi: pass migrate_data to recover_method if live migrate fails https://review.openstack.org/507874 | |
| 14:23:20 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146 | |
| 14:24:54 | openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements https://review.openstack.org/493187 | |
| 14:27:50 | mriedem | gibi: appears that test_live_migrate_delete has introduced a race http://logs.openstack.org/87/507687/2/check/gate-nova-tox-functional-ubuntu-xenial/90cc144/testr_results.html.gz | |
| 14:28:13 | mriedem | i was noticing some stuff like this yesterday when writing another test, | |
| 14:28:27 | mriedem | i noticed that we set migration and instance state before we're actually done cleaning up live migration things, | |
| 14:28:34 | mriedem | so tests that rely on asserting the cleanups can get racy | |
| 14:30:59 | gibi | mriedem: I have to check if we can wait for notification or instance action to avoid the race | |
| 14:31:37 | gibi | but I guess we need a bug so I can go and file it if you haven't already filed it | |
| 14:32:36 | mriedem | gibi: https://bugs.launchpad.net/nova/+bug/1719915 | |
| 14:32:37 | openstack | Launchpad bug 1719915 in OpenStack Compute (nova) "test_live_migrate_delete race fail when checking allocations: MismatchError: 2 != 1" [Medium,Confirmed] | |
| 14:34:30 | gibi | mriedem: thanks, I go and dig for a solution | |
| 14:36:22 | gibi | mriedem: btw, I will be mostly unavailable tomorrow and on Friday. Could you do the reporting about the notification subteam meeting on the weekly nova meeting? | |
| 14:36:33 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146 | |
| 14:37:00 | mriedem | gibi: sure | |
| 14:37:03 | gibi | mriedem: thanks a lot | |
| 14:38:06 | johnthetubaguy | mriedem: I was looking at the requesting traits in flavors, did we ever talk about requesting the absence of a trait (like request no CPU_FLAG_X available) | |
| 14:38:06 | openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements https://review.openstack.org/493187 | |
| 14:39:11 | efried | johnthetubaguy That's a fun thought. Is there a real use case for it? | |
| 14:39:35 | johnthetubaguy | efried: disable the VT CPU flag? | |
| 14:39:54 | johnthetubaguy | or disable hyperthreading | |
| 14:39:56 | dansmith | johnthetubaguy: not that I'm aware of, just required and preferred | |
| 14:40:33 | efried | johnthetubaguy I have been chastised already to make the distinction between "Ask for a resource that *can* do this thing" and "Ask for a resource and then switch this thing on". | |
| 14:41:22 | johnthetubaguy | I am happy to ignore all that for now | |
| 14:41:29 | johnthetubaguy | need the basic thing first | |
| 14:41:29 | efried | johnthetubaguy (or off). The latter thing definitely needs to be supported - just not via the placement/traits paths, I am led to understand. | |
| 14:42:16 | efried | I can't remember whether it was a review or IRC where jaypipes and I had that talk. Looking... | |
| 14:43:09 | efried | johnthetubaguy Whee: https://review.openstack.org/#/c/497713/4/specs/queens/approved/add-trait-support-in-allocation-candidates.rst@72 | |
| 14:43:24 | johnthetubaguy | cool, thanks, will have a read | |
| 14:44:14 | claudiub|3 | mriedem: cool, ty for the heads up. should be fine now. | |
| 14:44:46 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: PCI NUMA Policies https://review.openstack.org/361140 | |
| 14:44:55 | tssurya | dansmith, mriedem : so like the bug here https://bugs.launchpad.net/nova/+bug/1719487 even for the map instances, instead of querying with respect to the config file, we could extract the info from the cell_mappings of API database ? | |
| 14:44:57 | openstack | Launchpad bug 1719487 in OpenStack Compute (nova) "nova-manage db archive_deleted_rows is not multi-cell aware" [Wishlist,Triaged] - Assigned to Zhenyu Zheng (zhengzhenyu) | |
| 14:45:16 | dansmith | tssurya: yeah | |
| 14:47:21 | tssurya | dansmith : so to start working on this I would need a bug ? | |
| 14:47:53 | dansmith | tssurya: it's not really a bug, just an enhancement, but mriedem loves paperwork, so probably easiest to have one | |
| 14:48:50 | tssurya | dansmith : yes :D will open one then | |
| 14:49:15 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/ocata: Updated from global requirements https://review.openstack.org/490256 | |
| 14:51:03 | mriedem | note that was marked as wishlist | |
| 14:51:10 | mriedem | because yeah it's not really a bug | |
| 14:51:17 | mriedem | johnthetubaguy: idk, | |
| 14:51:33 | mriedem | logic operators on required and preferred traits is not something i'm thinking about | |
| 14:55:42 | cdent | not feels inevitable, but when the query parameters for allocation candidates becomes skynet, it’s not my fault | |
| 14:56:25 | tssurya | mriedem : yes I understand its not exactly a bug ; | |
| 15:00:43 | mriedem | i wonder if anyone has actually tried benchmarking and comparing ocata to pike for claims in the scheduler, and with multiple scheduler processes, since we claim that's a safe thing to do now https://docs.openstack.org/releasenotes/nova/pike.html#id2 | |
| 15:01:01 | mriedem | "The FilterScheduler driver now provides allocations to the Placement API, which helps concurrent schedulers to verify resource consumptions directly without waiting for compute services to ask for a reschedule in case of a race condition. That is an important performance improvement that includes allowing one to use more than one scheduler worker if there are capacity concerns. For more details, see the Pike Upgrade Notes fo | |
| 15:01:01 | mriedem | acement." | |
| 15:01:22 | dansmith | mriedem: so along those lines, I noticed that we are pegging the crap out of placement during scheduling 100 instances | |
| 15:01:24 | mriedem | dansmith: red hat has a perf lab right? is ^ something they have talked about? | |
| 15:01:38 | dansmith | even on my fast piece of hardware, placement pegs a CPU | |
| 15:01:49 | dansmith | mriedem: not that I've heard | |
| 15:01:58 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146 | |
| 15:02:13 | cdent | yeah, we had “do some performance testing” in the weekly rp update for so long that I eventually took it out from apparent lack of interest | |
| 15:02:38 | mriedem | ok. our public cloud guys have made tweaks to the scheduler for performance in mitaka, and lots of those tweaks i've said, "this thing in pike should resolve/replace that" but i don't have hard evidence | |
| 15:02:44 | cdent | It would surprise me not one tiny bit that it is not as performant as expected, because the only testing I’m aware of was done using mostly just a database, and not any of the other bits | |
| 15:02:52 | cdent | and since then the queries have modified quite a bit | |
| 15:02:57 | cdent | and we’ve added more objects | |
| 15:03:19 | mriedem | cdent: so i don't think you were around yesterday when we were talking about this, | |
| 15:03:32 | openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements https://review.openstack.org/493187 | |
| 15:03:38 | mriedem | but i realized, after several hours, yesterday why i couldn't burst 500 (fake) guest vms on my single node devstack | |
| 15:03:42 | mriedem | and they were all going novalidhost | |
| 15:03:43 | cdent | I had an afternoon in an attorney’s office ... | |
| 15:03:49 | dansmith | also keep in mind that a little slower scheduler performance compares very favorably to 10% retries in the background because we choose bad computes | |
| 15:04:04 | cdent | mriedem: what was the cause? | |
| 15:04:06 | mriedem | dansmith: that's why i'd want to compare ocata to pike | |
| 15:04:20 | mriedem | cdent: the ultimate cause was a 409 response from placement when putting the allocations during scheduling | |
| 15:04:24 | mriedem | we retry that 3 times, | |