Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-12-07
18:10:57 openstack bug 1799727 in OpenStack Compute (nova) rocky "CPU_Allocation_Ratio from nova.conf doesn't update exisiting providers" [High,In progress] https://launchpad.net/bugs/1799727 - Assigned to Matt Riedemann (mriedem)
18:10:57 openstackgerrit Merged openstack/nova stable/rocky: Add functional recreate test for bug 1799727 https://review.openstack.org/614563
18:11:03 openstackgerrit Merged openstack/nova stable/rocky: Provide allocation_ratio/reserved amounts from update_provider_tree() https://review.openstack.org/614564
18:11:10 openstack bug 1796737 in OpenStack Compute (nova) rocky "resize: hypervisor local_gb_used still reports usage even with volume-backed instances after fix for bug 1469179" [Medium,In progress] https://launchpad.net/bugs/1796737 - Assigned to Matt Riedemann (mriedem)
18:11:10 openstackgerrit Merged openstack/nova stable/rocky: Add regression test for bug 1796737 https://review.openstack.org/614587
18:11:15 openstackgerrit Merged openstack/nova stable/rocky: Properly track local root disk usage during moves https://review.openstack.org/614588
18:11:22 openstackgerrit Merged openstack/nova stable/rocky: Time how long select_destinations() takes in conductor https://review.openstack.org/608575
18:11:28 openstackgerrit Merged openstack/nova stable/rocky: Refix disk size during live migration with disk over-commit https://review.openstack.org/602477
18:11:35 openstackgerrit Merged openstack/nova stable/rocky: Handle IndexError in _populate_neutron_binding_profile https://review.openstack.org/610163
18:12:12 melwitt stephenfin: been meaning to ask you, the nova-next job stopped testing vnc console with TLS several months ago, see http://logs.openstack.org/92/621692/1/check/nova-next/28869ae/logs/screen-n-novnc-cell1.txt.gz#_Dec_04_01_01_59_749371 I tried digging into it and couldn't find what's going wrong, how it can't find the cert file. wanted to give you a heads up in case you had any clues
18:21:31 sean-k-mooney melwitt: i wonder if devstack change its default or something and its not enableing tls anymore
18:23:28 melwitt sean-k-mooney: yeah, it's supposed to set everything up so long as the tls-proxy service is enabled https://github.com/openstack-dev/devstack/blob/78a564bb0304b6f930e1491e7e116a0a0f6d9ab6/stack.sh#L848
18:24:21 melwitt and in the example log I linked, it is enabled http://logs.openstack.org/92/621692/1/check/nova-next/28869ae/logs/devstacklog.txt.gz#_2018-12-04_00_45_46_475
18:24:55 melwitt I looked around devstack months ago looking for a change but didn't find anything. I could have missed it
18:53:37 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP Reject migration requests when src compute is down https://review.openstack.org/623489
18:57:14 openstackgerrit Jay Pipes proposed openstack/nova master: add InstanceList.get_all_uuids_by_hosts() method https://review.openstack.org/623557
18:57:15 openstackgerrit Jay Pipes proposed openstack/nova master: single pass instance info fetch in host manager https://review.openstack.org/623558
18:58:25 leakypipes belmoreira: ^^ would be great to get some initial feedback from you on these performance patches to the scheduler. See commit message from https://review.openstack.org/#/c/623558/ for details.
18:58:30 leakypipes melwitt: ^^
19:00:39 melwitt kewl
19:02:41 openstackgerrit Jack Ding proposed openstack/nova master: Preserve UEFI NVRAM variable store https://review.openstack.org/621646
19:09:14 leakypipes topic name, that is...
19:09:42 mriedem call me ishaeml
19:09:46 mriedem *ishmael
19:11:27 mriedem dansmith: you may enjoy https://review.openstack.org/#/c/615347 and it also resolves a functional race bug
19:11:32 mriedem http://status.openstack.org/elastic-recheck/#1800472
19:39:49 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Drop nova-multiattach job https://review.openstack.org/623568
19:45:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Drop nova-multiattach job https://review.openstack.org/606981
19:48:29 openstackgerrit Ken'ichi Ohmichi proposed openstack/nova master: Add descriptions about microversions https://review.openstack.org/619144
19:57:32 cfriesen is there an equivalent to "nova service-list" in OSC, which would show the state of each compute node's service separately?
19:58:27 cfriesen never mind, found "openstack compute service list"
20:10:51 dansmith mriedem: see comment
20:10:58 dansmith +3 on the code part
20:18:38 radez dansmith: thx, we're working on multi-tenancy with ironic trying to test using networking-ansible to isolate networks
20:19:06 radez this works fine, though in our CI we're setting up two tenants and trying to deploy an instance using ironic in each of the tenants
20:19:09 dansmith radez: on master or rocky?
20:19:15 radez rocky
20:19:28 radez when we schedule the first node it deploys just fine
20:19:41 dansmith okay just to be clear.. nova with compute_driver=ironic, right?
20:20:12 radez yes i believe so , let me dousble check that, I've been relying on ooo to setup ironic for me
20:21:04 dansmith just want to make it clear you're using nova to schedule an instance that will be running on a node in ironic.. the way you worded it above made me first think you were just talking about standalone ironic
20:22:14 radez sry, you're correct we're using nova to schedule and ironic node
20:22:23 dansmith okay, sorry, continue :)
20:25:21 radez gathering my details here to try and depict this correctly
20:25:46 radez so we deploy the first node and that goes swimmingly. comes active we can connect to it.
20:25:52 radez this is in one tenant
20:26:40 radez in the second tenant we try and deploy another node and the nova logs apprear to indicate that when it goes to get available nodes to deploy to it returns 1 node but it returns the already allocated node instead of the available node
20:27:01 radez lemme gather a few logs that depict this
20:27:07 dansmith are you sure it's selecting the same *node* and not the same nova-compute?
20:27:35 dansmith because if you have only one machine running nova-compute, that compute service will be responsible for all the ironic nodes
20:28:14 dansmith I assume that's not the problem because you would likely not even notice and not be here with an issue, but just ... in case it's relevant
20:28:46 dansmith also, while you're collecting logs, this shouldn't have anything to do with the multi-tenancy part, as the scheduling, picking a node, reserving a node, etc doesn't really factor tenants into the calculus
20:29:06 radez I *think* that's not the case because the ironic node uuids and the nova instance uuids match up in the logs
20:29:35 radez fair enough, just laying out the context incase that was helpful
20:29:47 dansmith okay, they should only really line up in the compute log I think, probably not in the scheduling log, fwiw
20:29:53 dansmith ack, yep, just letting you know
20:31:25 dansmith radez: my tacos are just about warmed, so keep collecting logs and dumping info here and I'll reply when my fingers are clean :)
20:31:44 dansmith also, mriedem is here, he's real smart, and he was just telling me he was bored and looking for a good goose to chase :P
20:32:24 radez lol, cool I just got a call I need to take that will distract me for about 30 mins. I'll finish collecting my logs and ping one of you in a bit, sry about that.... I appreciate your help!
20:33:17 mriedem what's good for the goose is good for the gander
20:34:00 dansmith radez: okay, also keep in mind it's friday afternoon so probably not many hours left in the day (and I'm sure you're ahead of the rest of us in EST) :D
20:42:01 dansmith taco consumption complete
20:43:01 mriedem dansmith: replied on the reno thing - i'm ok with dropping that if it would just cause more concern than it's worth
20:43:18 mriedem that compat was really kind of out the window since pike i think
20:43:21 mriedem when the api was made cells aware
20:43:59 dansmith mriedem: yeah, so I'd just drop it if you're cool with it
20:44:13 dansmith hah
20:44:23 mriedem i'm very cool
20:44:24 dansmith zuul says 135h remaining on my patch I submitted at like 7am this morning
20:45:04 mriedem https://review.openstack.org/#/c/602804/ queued for at least 36 hours, hit a job timeout
20:46:36 dansmith good lord
20:46:57 dansmith maybe we should start enacting a french work-week to avoid overloading the gate
20:46:58 openstackgerrit Matt Riedemann proposed openstack/nova master: Drop pre-cellsv2 compat in compute API.get() https://review.openstack.org/615347
20:46:59 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove "API Service Version" upgrade check https://review.openstack.org/615348
21:01:53 radez ok, I'm back now, sry about that
21:02:00 radez dansmith: mriedem: http://paste.openstack.org/show/736843/
21:02:25 radez this shows the uuids of the bm nodes and the nova instances that seems to be behaving like this
21:03:20 radez and the nova logs where nova seems to assign the already assigned bm node to the new nova instance
21:03:51 radez let me know if there's more logs I can get to help
21:04:41 dansmith radez: so these nodes should all exist in placement with a single inventory item for the resource/node class,
21:04:51 dansmith which means they should never get past the scheduler with two instances on the same node
21:05:12 radez translation was helpful thx :)
21:05:31 radez [instance: a840afbc-1314-44d6-83b3-c20b790b9322] Claim successful on node 3c3eb5ee-a358-421a-b2e9-7fd9a017ba13
21:06:34 radez hm, well now I think I've given you the logs that show that the associated ironic node and nova instance are properly associated together
21:07:42 mriedem do the flavors have cpu/ram/disk set to anything !0?
21:07:44 radez oh wait, sry I'm getting my self confused... right so does that log line I just pasted here show that nova is trying to
21:07:46 dansmith radez: what log is this? just nova compute?
21:08:06 radez assign ironic node 3c3eb5ee to nova instance a840afbc?
21:08:34 radez yea nova-compute log
21:08:43 dansmith radez: so probably want to look at the scheduler log
21:09:00 dansmith radez: scheduler is what decides which node it is going to tell compute to use for each instance
21:10:10 radez k lemme find a840afbc in the schedule logs
21:13:17 radez http://paste.openstack.org/show/736844/
21:13:43 openstackgerrit Merged openstack/os-vif master: add isolate_vif config option https://review.openstack.org/612534
21:13:46 radez that also shows a840afbc trying to select 3c3eb5ee
21:14:32 radez hypervisor list incase that's helpful? http://paste.openstack.org/show/736845/
21:14:33 melwitt radez: what version of rocky is this? we had a bug around ironic inventory update in the middle of rocky https://review.openstack.org/593678 but as long as you have 18.0.0.0rc2 or later, you should have the fix for that
21:14:35 melwitt er, at the end of rocky
21:14:42 dansmith radez: can you show more of the log so we can see both?
21:15:07 radez yea, that was just a quick grep, lemme grab more of the scheduler log and check my version

Earlier   Later