| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-29 | |||
| 15:38:33 | mriedem | i'm bringing my white noise machine either way | |
| 15:38:47 | jroll | that's an odd name for your child | |
| 15:38:48 | tssurya | :( | |
| 15:38:52 | dansmith | tssurya: last year my room had a hole in the wall because they weren't finished with the remodel | |
| 15:39:00 | tssurya | really ?! | |
| 15:39:20 | tssurya | so its not really gonna like the Dublin PTG then.. | |
| 15:39:20 | dansmith | tssurya: the foundation got a *smashing* deal.. | |
| 15:39:34 | dansmith | tssurya: probably less snow, but no promises | |
| 15:39:49 | jroll | I missed the last one in denver, dansmith is making me so excited | |
| 15:39:55 | tssurya | heh, | |
| 15:40:04 | dansmith | jroll: HOOOOOOOOOONK | |
| 15:40:14 | dansmith | jroll: HOOOOOOOOOOOOOOOOOOOOOOOOOOOONK | |
| 15:40:22 | jroll | :P | |
| 15:40:23 | dansmith | every ten minutes | |
| 15:40:27 | dansmith | every | |
| 15:40:27 | dansmith | ten | |
| 15:40:29 | dansmith | minutes | |
| 15:40:35 | tssurya | I will be there on Thursday, so I have like 4 days to go around, maybe I can see the nice part of colorado | |
| 15:40:37 | dansmith | sometimes at night I still hear it | |
| 15:41:05 | dansmith | jroll: you know it's bad when at checkin they hand you a noise machine and earplugs | |
| 15:41:09 | dansmith | literally. | |
| 15:41:17 | dansmith | until they ran out of course | |
| 15:41:21 | jroll | yeah, so I heard | |
| 15:41:22 | edleafe | tssurya: Rocky Mountain National Park may help you if you're missing the Alps | |
| 15:41:36 | jroll | I get in monday night. so. I guess I'll bring my own | |
| 15:42:00 | tssurya | edleafe: nice I will see if I can go there, just waiting for the visa to be in my hand.. | |
| 15:42:31 | mriedem | tssurya: you're in luck! https://www.google.com/search?q=denver+broncos+schedule&ie=utf-8&oe=utf-8&client=firefox-b-1-ab#sie=m;/g/11hdb0xw6z;6;/m/059yj;dt;fp;1 | |
| 15:42:49 | mriedem | you can go watch the broncos get destroyed at home! | |
| 15:43:04 | tssurya | mriedem: ha! | |
| 15:43:22 | mriedem | https://deadspin.com/why-your-team-sucks-2018-denver-broncos-1828078886 | |
| 15:46:00 | cfriesen | mriedem: the "four friends kitchen" in Denver is really good, but a bit further away. | |
| 15:54:28 | mriedem | gibi: i think i know what's going on in https://bugs.launchpad.net/nova/+bug/1781648 | |
| 15:54:28 | openstack | Launchpad bug 1781648 in OpenStack Compute (nova) "heal_allocations test randomly failing with "ValueError: Field `compute_node_uuid' cannot be None"" [Medium,Confirmed] | |
| 15:54:40 | mriedem | we're racing with the caching scheduler starting up and initializing the cache of compute nodes | |
| 15:54:50 | mriedem | so by the time we hit the scheduler, the cache is empty | |
| 15:56:07 | gibi | mriedem: that sounds like a reasonable explanation. I guess I missunderstood the logs when I stated in the bug that the host creation overlaps with the scheduling | |
| 15:59:07 | gibi | mriedem: so I guess the fix is that we need to wait in the test a bit | |
| 16:00:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Restart scheduler in TestNovaManagePlacementHealAllocations https://review.openstack.org/597571 | |
| 16:00:45 | mriedem | sort of ^ | |
| 16:02:09 | mriedem | i thought about also changing this https://github.com/openstack/nova/blob/master/nova/scheduler/caching_scheduler.py#L78 to "if not self.all_host_states:" to handle the empty list case, but that could still be a weird race failure if we have 1 host but not both in the cache | |
| 16:02:23 | gibi | mriedem: your proposed start / stop is way better than a simple sleep in the test | |
| 16:02:34 | mriedem | i learned it from watching you gibi | |
| 16:04:09 | gibi | :) | |
| 16:05:32 | melwitt | . | |
| 16:05:51 | gibi | mriedem: I'm wondering how many other functional tests uses CachingScheduler and therefore might be affected by the same race | |
| 16:06:10 | mriedem | gibi: some do, but they might start the scheduler after the computes, i was just starting to audit that | |
| 16:06:43 | gibi | mriedem: cool | |
| 16:14:37 | mriedem | mlavalle: smcginnis: on this cross-cell cold migration thing nova will definitely need some pre-validation of the selected host, | |
| 16:15:04 | mriedem | i'm wondering if simply trying to create a volume attachment/port binding on the selected target host *in another cell* would be enough to tell us if that's not going to work for storage/networking | |
| 16:15:20 | mriedem | or in the case of port bindings, will neutron only fuss when we try to activate the target host port binding? | |
| 16:16:26 | jaypipes | zigo, mriedem: we have a rough timeframe of when this began occurring (the allocation ratio 0 thing...)? | |
| 16:17:28 | mriedem | earliest i know of is when naichuans reported it in the ML | |
| 16:18:01 | mlavalle | mriedem: the port binding is the result of asking the mechanism managers if they can bind the port in the indicated host. so the binding process is what you call the pre-validation of the selected host | |
| 16:18:11 | melwitt | dansmith: +1 to meeting skippage | |
| 16:18:12 | mlavalle | mechanism drivers^^^^ | |
| 16:19:38 | mlavalle | so when you say create_port_binding, you are asking the mechanism drivers whether any of them can bind the port in the designated host | |
| 16:20:02 | mriedem | mlavalle: and if the network that the port is in doesn't cover that dest host, the port binding creation should fail? | |
| 16:20:48 | sean-k-mooney | mriedem: it will fail of non of the ml2 driver can bind the port to the correct network | |
| 16:20:53 | mlavalle | mriedem: correct. the question that the drivers ask themselves is whether they can a segment of the network reachable in that host | |
| 16:21:06 | mlavalle | they have a segment^^^^ | |
| 16:21:28 | melwitt | can anyone confirm whether PCI and SRIOV PCI devices are treated the same as far as the scheduler is concerned? | |
| 16:21:54 | sean-k-mooney | mlavalle: that segment is only relevent for provider networks with the multi segment network extention correct normmal it 1 segment per network | |
| 16:22:48 | sean-k-mooney | mlavalle: as in in the pci filter? | |
| 16:22:59 | sean-k-mooney | melwitt: ^ | |
| 16:23:53 | sean-k-mooney | melwitt: they are more or less treated the same but they do have a different type in the pci manager (type-pci for passthough and type-PF or type-VF for sriov) | |
| 16:23:57 | melwitt | sean-k-mooney: maybe? just in general, for NUMA scheduling, if SRIOV PCI devices are treated differently at all or no | |
| 16:23:59 | mlavalle | sean-k-mooney: but regardless, today we can bind a port in multiple cells deployments, right? | |
| 16:24:14 | melwitt | sean-k-mooney: I see, thank you | |
| 16:25:00 | sean-k-mooney | melwitt: from a numa perspctive they are treated the same. we store the numa node info the same way in the db | |
| 16:25:20 | sean-k-mooney | mlavalle: yes i think so but never tried it | |
| 16:26:38 | mlavalle | sean-k-mooney: so that means that the mechanism drivers are being able to answer the question, can I bind a port here or not? in multiple port bindings, we are asking that same question, it's just that the port is also bound somewhere else | |
| 16:26:46 | sean-k-mooney | mlavalle: cells is a nova only ting. neutron has availablity zones. if you had a different neutron availableity zone per cell can a netowrk span neutron availablity zones | |
| 16:27:53 | mriedem | i would think, | |
| 16:28:02 | sean-k-mooney | mlavalle: well we are talking about cross cell cold migration so jsut like the live migration case we will have multiple port bindings so from a neutron point of view it should be identical | |
| 16:28:03 | mriedem | when we have an attached port, bound to the source host, | |
| 16:28:09 | mriedem | nova puts the az on the port binding information, | |
| 16:28:19 | mriedem | and neutron would be relying on that to know if we can bind to another host in another a | |
| 16:28:21 | mriedem | *az | |
| 16:28:24 | mlavalle | sean-k-mooney: that is what I say | |
| 16:28:24 | jaypipes | mriedem: I have a sneaking suspicion this patch is the cause: https://github.com/openstack/nova/commit/c9b74bcfa09d11c2046ce1bfb6dd8463b3a2f3b0 | |
| 16:28:37 | mriedem | jaypipes: that's only in master | |
| 16:28:41 | mriedem | zigo is hitting this in rocky | |
| 16:28:49 | sean-k-mooney | mlavalle: yep i am agreeing :) | |
| 16:28:59 | mlavalle | sean-k-mooney: LOL | |
| 16:29:16 | mriedem | btw, is it strange we don't have a cross_az_attach=False thing for nova/neutron like we have for nova/cinder? | |
| 16:29:17 | mlavalle | sean-k-mooney: will you be in Denver? | |
| 16:29:21 | sean-k-mooney | mriedem: neutron does not use the nova AZ as part of port bininging just the hostname | |
| 16:29:40 | sean-k-mooney | mlavalle: yes i will | |
| 16:29:43 | mriedem | so why do we set the device_owner on the port? | |
| 16:29:46 | mlavalle | Great! | |
| 16:29:47 | mriedem | using the instance az? | |
| 16:30:42 | mlavalle | because you want to keep track of where it is, on the Nova side, but I am just speculating | |
| 16:30:43 | sean-k-mooney | mriedem: that is a good question to which i do not know the answer | |
| 16:30:54 | mriedem | it might be used by the neutron callback code | |
| 16:31:10 | jaypipes | mriedem: oh... I thought you said this only recently occurred.. | |
| 16:31:26 | mriedem | jaypipes: it is only recently reported | |
| 16:31:35 | mriedem | but i also thought about that _update change but it's master only | |
| 16:31:39 | mriedem | and zigo said he's hitting it on rocky | |
| 16:31:55 | sean-k-mooney | mriedem: im pretty sure its not used by neutron at all. my geuess is this might be from nova networks and we just kept doing it | |
| 16:32:08 | sean-k-mooney | the AZ that is | |