| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-09 | |||
| 18:56:58 | cdent | ah, so doubling gets cleaned up too soon (until later in the series)? | |
| 18:56:58 | mriedem | Sending updated allocation [{'resource_provider': {'uuid': '97ce076a-2644-465b-95dc-fc0674152976'}, 'resources': {u'VCPU': 0, u'MEMORY_MB': 0, 'DISK_GB': 0}}] for instance 6fa7b953-c1fe-4520-8564-aeba8e90aece after removing resources for 97ce076a-2644-465b-95dc-fc0674152976. {{(pid=19567) remove_provider_from_instance_allocation /opt/stack/new/nova/nova/scheduler/client/report.py:1091}} | |
| 18:57:12 | mriedem | oh nvm | |
| 18:57:17 | mriedem | that's known, i was thinking that was the bottom change | |
| 18:59:17 | mriedem | the current_allocs get logged but they don't have 2 VCPU | |
| 18:59:20 | jaypipes | dansmith: are you actively making changes to these patches? if not, I can address mriedem's comments. | |
| 18:59:26 | mriedem | so the doubling isn't happening it appears | |
| 19:00:43 | mriedem | this is the resize in the scheduler http://logs.openstack.org/10/488510/30/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/5a36c66/logs/screen-n-sch.txt.gz#_Aug_09_16_27_37_363471 | |
| 19:00:53 | mriedem | req-7cbe9eeb-dc55-4986-be30-99bd7a03d2c4 | |
| 19:01:12 | openstackgerrit | Merged openstack/nova master: doc: provide more details on scheduling with placement https://review.openstack.org/491900 | |
| 19:01:49 | mriedem | req-7cbe9eeb-dc55-4986-be30-99bd7a03d2c4 | |
| 19:01:51 | mriedem | oops | |
| 19:01:56 | mriedem | Aug 09 16:27:37.633395 ubuntu-xenial-internap-mtl01-10347573 nova-scheduler[17280]: DEBUG nova.scheduler.client.report [None req-7cbe9eeb-dc55-4986-be30-99bd7a03d2c4 tempest-MigrationsAdminTest-402050297 tempest-MigrationsAdminTest-402050297] Doubling-up allocation request for move operation. {{(pid=17280) _move_operation_alloc_request /opt/stack/new/nova/nova/scheduler/client/report.py:162}} | |
| 19:02:01 | mriedem | it says it's double stuffing it | |
| 19:02:05 | openstackgerrit | Merged openstack/nova master: Add a prelude section for Pike https://review.openstack.org/491424 | |
| 19:02:19 | mriedem | and it looks like it does | |
| 19:02:20 | mriedem | {u'VCPU': 2, u'MEMORY_MB': 192}}]} {{(pid=17280) _move_operation_alloc_request /opt/stack/new/nova/nova/scheduler/client/report.py:202}} | |
| 19:02:20 | mriedem | Aug 09 16:27:37.633725 ubuntu-xenial-internap-mtl01-10347573 nova-scheduler[17280]: DEBUG nova.scheduler.client.report [None req-7cbe9eeb-dc55-4986-be30-99bd7a03d2c4 tempest-MigrationsAdminTest-402050297 tempest-MigrationsAdminTest-402050297] New allocation request containing both source and destination hosts in move operation: {'allocations': [{'resource_provider': {'uuid': u'97ce076a-2644-465b-95dc-fc0674152976'}, 'resource | |
| 19:02:58 | openstackgerrit | Merged openstack/nova master: Mark max microversion for Pike in history doc https://review.openstack.org/491581 | |
| 19:03:10 | mriedem | 74152976'}, u'resources': {u'VCPU': 1, u'MEMORY_MB': 128}}]} {{(pid=17280) _claim_resources /opt/stack/new/nova/nova/scheduler/filter_scheduler.py:289}} | |
| 19:03:10 | mriedem | Aug 09 16:27:37.713717 ubuntu-xenial-internap-mtl01-10347573 nova-scheduler[17280]: DEBUG nova.scheduler.filter_scheduler [None req-7cbe9eeb-dc55-4986-be30-99bd7a03d2c4 tempest-MigrationsAdminTest-402050297 tempest-MigrationsAdminTest-402050297] Successfully claimed resources for instance 6fa7b953-c1fe-4520-8564-aeba8e90aece using allocation request {u'allocations': [{u'resource_provider': {u'uuid': u'97ce076a-2644-465b-95dc- | |
| 19:03:10 | mriedem | but then i see this | |
| 19:03:22 | mriedem | wtf | |
| 19:03:22 | mriedem | which is the single allocation again | |
| 19:03:42 | openstackgerrit | Merged openstack/nova master: Document service layout for consoles with cells https://review.openstack.org/491914 | |
| 19:03:45 | mriedem | OHHHHHHHHH | |
| 19:03:47 | mriedem | i know why | |
| 19:04:02 | mriedem | mfing pass by reference | |
| 19:04:14 | openstackgerrit | Merged openstack/nova master: Cleanup release note about ignoring allow_same_net_traffic https://review.openstack.org/491855 | |
| 19:04:22 | jaypipes | mriedem: that's passing by reference. | |
| 19:04:25 | mriedem | dansmith: jaypipes: cdent: problem is right here https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L198 | |
| 19:04:31 | mriedem | return new_alloc_req | |
| 19:04:36 | mriedem | but we're not touching new_alloc_req | |
| 19:04:52 | mriedem | so we double stuff but don't persist | |
| 19:05:07 | jaypipes | yep | |
| 19:05:54 | cdent | oops | |
| 19:06:25 | cdent | does any of gibi’s pending functional stuff cover that? | |
| 19:06:35 | mriedem | not sure | |
| 19:06:45 | jaypipes | func or unit should have covered this... | |
| 19:06:50 | mriedem | anyway, i'll get this nova-status thing done | |
| 19:06:58 | mriedem | and then could push a fix for that on top | |
| 19:07:01 | jaypipes | mriedem: k, I'll fix the above. | |
| 19:07:02 | mriedem | and then we rebase the series on top of those | |
| 19:07:04 | mriedem | or that | |
| 19:07:13 | jaypipes | mriedem: you tell me. what do you want? | |
| 19:07:38 | cdent | I really need to go, but please, if stuff’s not stable by the time you guys expire tonight, let me know so I can continue it in the morning | |
| 19:07:45 | jaypipes | mriedem: I can tack on a fix on the bottom of this series? | |
| 19:07:54 | mriedem | jaypipes: yes it would have to be | |
| 19:08:08 | mriedem | note i've got other issues in that 2nd change though, | |
| 19:08:12 | mriedem | which i guess could be follow ups | |
| 19:08:22 | mriedem | i'm just kind of getting tired of making myself TODOs about chasing follow ups | |
| 19:08:25 | mriedem | but realize we're short on time | |
| 19:08:58 | jaypipes | mriedem: I'm happy to address your comments on the second patch. just need to know if dansmith is actively working on these. | |
| 19:09:08 | dansmith | nope | |
| 19:09:44 | jaypipes | dansmith: gotcha. I'll get on this then. | |
| 19:10:37 | cdent | k, in that case I’ll look at the existing main stack in the morning and see where we’re at | |
| 19:10:40 | cdent | g’night | |
| 19:10:46 | mriedem | o/ | |
| 19:11:19 | mriedem | so i think unit tests are covering _move_operation_alloc_request | |
| 19:11:27 | mriedem | New allocation request containing both source and destination hosts in move operation: {'allocations': [{'resource_provider': {'uuid': u'97ce076a-2644-465b-95dc-fc0674152976'}, 'resources': {u'VCPU': 2, u'MEMORY_MB': 192}}]} | |
| 19:11:30 | mriedem | that's from that same method, | |
| 19:11:34 | mriedem | after merging the allocations | |
| 19:11:37 | mriedem | by reference | |
| 19:12:43 | mriedem | ah a red herring | |
| 19:12:53 | mriedem | Successfully claimed resources for instance 6fa7b953-c1fe-4520-8564-aeba8e90aece using allocation request {u'allocations': [{u'resource_provider': {u'uuid': u'97ce076a-2644-465b-95dc-fc0674152976'}, u'resources': {u'VCPU': 1, u'MEMORY_MB': 128}}]} {{(pid=17280) _claim_resources /opt/stack/new/nova/nova/scheduler/filter_scheduler.py:289}} | |
| 19:12:56 | mriedem | ^ is stale | |
| 19:12:56 | jaypipes | not for same-host-resize, though, right? | |
| 19:13:11 | mriedem | no we doubled correctly | |
| 19:13:13 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L201 | |
| 19:13:20 | mriedem | New allocation request containing both source and destination hosts in move operation: {'allocations': [{'resource_provider': {'uuid': u'97ce076a-2644-465b-95dc-fc0674152976'}, 'resources': {u'VCPU': 2, u'MEMORY_MB': 192}}]} | |
| 19:13:22 | mriedem | that's doubled | |
| 19:13:45 | mriedem | what we're logging here: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L293 | |
| 19:13:47 | mriedem | is stale | |
| 19:13:50 | mriedem | that's why i got confused | |
| 19:14:17 | mriedem | however, when we get to the RT, and subtract, we lost the doubled allocation somewhere still | |
| 19:14:33 | mriedem | jaypipes: so i don't know what the bug is yet | |
| 19:14:37 | mriedem | we just have really misleading logging | |
| 19:15:01 | jaypipes | mriedem: I don't agree that the above line is "stale". | |
| 19:15:09 | mriedem | of course it is | |
| 19:15:15 | mriedem | alloc_req comes from placement | |
| 19:15:18 | mriedem | we double it | |
| 19:15:27 | mriedem | then we log the original alloc_req from placement | |
| 19:15:29 | mriedem | which is not double | |
| 19:15:50 | mriedem | follow the 4 log messages starting here http://logs.openstack.org/10/488510/30/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/5a36c66/logs/screen-n-sch.txt.gz#_Aug_09_16_27_37_563749 | |
| 19:17:02 | jaypipes | mriedem: yeah, right. | |
| 19:17:30 | mriedem | so, we're still losing the doubled allocation somewhere | |
| 19:17:33 | mriedem | my guess is the RT is overwriting it? | |
| 19:17:46 | jaypipes | mriedem: I will change placement client claim_resources() to return the possibly-updated-for-move-operation alloc_req. | |
| 19:18:16 | mriedem | jaypipes: we don't have to change all of that, just drop alloc_req from this log message https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L293 | |
| 19:18:25 | mriedem | we already logged the original and new thing | |
| 19:18:36 | jaypipes | k | |
| 19:18:38 | mriedem | well, we logged the new thing | |
| 19:23:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Require Placement 1.0 in nova-status upgrade check https://review.openstack.org/492234 | |
| 19:25:38 | mriedem | dansmith: ^ thar she blar | |
| 19:26:11 | sdague | mriedem: commit message? | |
| 19:26:23 | sdague | 1.10 right? | |
| 19:26:27 | dansmith | mriedem: I got distracted by a rant opportunity.. did you figure out the doubling thing? | |
| 19:26:38 | mriedem | crap | |
| 19:26:42 | jaypipes | dansmith: still working on it. | |
| 19:27:06 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Require Placement 1.10 in nova-status upgrade check https://review.openstack.org/492234 | |