Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-20
16:32:09 mnaser sean-k-mooney: ok that makes sense then, i will try to reset the infinity fabric speed down to see if the machine stops crashing :\
16:34:28 Sundar sean-k-mooney: Thanks a lot for reviewing the Cyborg spec despite your busy schedule. All the best.
16:40:15 mriedem gmann: the only one passing is stable/train
16:42:30 efried mriedem: it sounds like if that one merges it might fix master (I still don't understand how though)
16:42:59 gmann mriedem: yeah some unrelated failure. i did recheck on stein. after train cherry pick merged we can check if problem is sovled. otherwise need to get all in
16:43:30 gmann it all depends on how zuul pick the job definition due to those branche variant.
16:44:16 gmann we can try one by one. last option to avoid barbican gate is to make grenade job as py2 explicitly.
16:44:25 gmann *gate break
16:47:01 mriedem efried: because the job definitions are branch specific and grenade deploys the old side using stable/train and the new side using master
16:47:17 mriedem so sometimes to fix a thing on master for grenade you need to do something in master-1
16:47:48 mnaser sean-k-mooney: ok making good progress, according to numastat it ended up scheduling to numa cell0 and cell1 which means this is the most "optimized" setup
16:48:22 mnaser topology is 8 sockets, i wonder if it would be better to model the topology as "2 sockets, 4 cores, 1 threads" or "2 sockets, 2 cores, 2 threads" too
16:56:19 efried dansmith: are there any operations other than shelve for which the swift blob would be appropriate?
16:57:15 dansmith well, what are you going to do about snapshot?
16:57:23 dansmith snapshot gets a fresh tpm?
16:57:34 dansmith that doesn't work well for people that use snapshot as a backup mechanism
16:57:42 dansmith speaking of that, the backup api :)
16:58:48 mnaser burn it with fire
16:58:54 efried dansmith: gr, config option? save_tpm_with_snapshot_note_does_not_apply_to_shelve_where_we_always_do_that_anyway = $bool
16:59:07 dansmith idk
16:59:18 efried open question I guess, sigh
16:59:19 dansmith I mean, not a config option, no
16:59:45 dansmith but not sure what the answer is
16:59:53 sean-k-mooney mnaser: that is becasue we use itertool.permuations so we will initally iterate over the numa nodes in order trying to fit the vitual numa node to the host
16:59:54 efried I think the answer has to be no
17:00:05 efried because we have no way to associate in that scenario
17:00:12 dansmith if I snapshot my instance and plan to roll ack if my OS upgrade fails, I'm going to be super surprised that I can't read my encrypted data volume (ever again)
17:00:15 efried because we're relying on the instance sysmeta to associate for shelve.
17:00:18 sean-k-mooney mnaser: so in that specific case it will work
17:00:19 mnaser sean-k-mooney: so maybe eventually machines might get some inefficent assignments
17:00:24 sean-k-mooney mnaser: but in general it wont
17:00:30 sean-k-mooney mnaser: yes
17:00:40 mnaser so its possible i hit node4+node5 which is a bad time
17:00:50 dansmith efried: I think it'd be good to get some jroll input on this
17:00:57 efried agree
17:01:01 sean-k-mooney if node 0,2,3 could fit it it would select 0 and 2
17:01:06 sean-k-mooney and never check 3
17:01:09 dansmith efried: but I think people will be pretty surprised if it's gone
17:01:15 mnaser sean-k-mooney: so far im seeing some numbers increasing by 25% -- and ok i see
17:01:42 efried dansmith: but then we go back to needing the association in the instance meta, because there's nothing else
17:02:20 sean-k-mooney mnaser: that is why i said we liekly could optimise this using the numa distance but its more complex then what we do today
17:02:29 dansmith efried: image meta you mean? I'm not saying we should, I'm just poking holes in it which.. if people lose data they get mad
17:02:30 sean-k-mooney mnaser: and it has implciations for modeling in plamcent
17:02:33 dansmith efried: I'm on a call right now though
17:02:36 efried sorry, yeah, image meta.
17:02:37 mnaser sean-k-mooney: yes that sounds massively complex to get implemented -- but would be so awesome
17:03:12 sean-k-mooney mnaser: what feature that relates to numa isint :)
17:03:18 mnaser hah, good one
17:03:30 mnaser sean-k-mooney: i mean in my case, the fact memory doesnt float across 8/16 numa nodes, at least 2 is better, the only "bad" potential idea is if a VM get pinned to two NUMA nodes that are pretty far apart
17:03:58 sean-k-mooney right as in on different sockets.
17:04:07 mnaser yep
17:04:17 sean-k-mooney but even then its still better then floating as it will be deterministic
17:04:53 sean-k-mooney well in that once the instnace i booted its performance should not change
17:17:50 openstackgerrit Merged openstack/nova stable/train: Imported Translations from Zanata https://review.opendev.org/694911
17:32:45 openstackgerrit Boris Bobrov proposed openstack/nova master: Create a controller for qga when SEV is used https://review.opendev.org/693072
17:34:27 KeithMnemonic any change to please get some reviews on the "marker" patches mriedem was working on ? https://review.opendev.org/#/c/690721/4 Thanks!
17:36:59 dustinc efried, gibi, dansmith: RE Provider Config: Just got caught up on this morning's convo. Am I right in seeing that you guys want to A) drop NAME as identification method, and B) Allow both $UUID and $COMPUTE_NODE to identify same RP but $UUID takes precedence? If so I am 100% on board.
18:06:45 dansmith dustinc: I dunno about a conflict between a fixed uuid and $compute_node, but I would recommend handling that the same way I suggested for the name/uuid conflict, which is to log error and ignore both
18:33:04 openstackgerrit Boris Bobrov proposed openstack/nova master: Also enable iommu for virtio controllers and video in libvirt https://review.opendev.org/684825
18:33:04 openstackgerrit Boris Bobrov proposed openstack/nova master: Create a controller for qga when SEV is used https://review.opendev.org/693072
18:49:19 sean-k-mooney by the way i just looked at the implemented spec folder for train. looks like we have not moved them https://github.com/openstack/nova-specs/tree/master/specs/train/implemented
19:09:08 mriedem sean-k-mooney: you can run this and post the results https://github.com/openstack/nova-specs/blob/master/tox.ini#L39
19:14:48 sean-k-mooney sure just leaving to grab dinner but ill give it a try when i get back
19:15:49 sean-k-mooney ok so that checks with launchpad to determin if they were finsihed and then updates them
19:16:03 sean-k-mooney cool
19:54:26 openstackgerrit Mark Goddard proposed openstack/nova master: Clear rebalanced compute nodes from resource tracker https://review.opendev.org/695187
19:54:27 openstackgerrit Mark Goddard proposed openstack/nova master: Invalidate provider tree when compute node disappears https://review.opendev.org/695188
19:54:27 openstackgerrit Mark Goddard proposed openstack/nova master: Prevent deletion of a compute node belonging to another host https://review.opendev.org/694802
19:54:28 openstackgerrit Mark Goddard proposed openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/695189
20:38:41 efried dustinc: The way I suggested yesterday still works IMO, and it requires less churn in both design and already-written code.
20:40:13 efried With dansmith's addendum
20:41:29 efried Which is as follows:
20:41:29 efried - Index by identifier (name *or* uuid) and fail hard on conflicts there
20:41:29 efried - Treat $COMPUTE_NODE as the default and a specific name/uuid as override
20:41:29 efried - If a provider appears such that you do find a conflicting name+uuid, log an error and ignore that provider completely. (This should only be possible for *nested* providers.)
21:00:15 efried dansmith: shelve without offload only works for non-volume-backed, right?
21:01:53 efried and followup question, should we do the swift thing for shelve without offload, or only with offload? I guess the question is a) does unshelve always happen on same host in that case; b) do we care about "shelving" the minimal amount of storage required by the vdev?
21:02:14 artom sean-k-mooney, as promised, if you still have energy today, https://review.opendev.org/#/c/691062/ is ready again
21:02:23 efried probably yes to b), just for the sake of form, so shelving the vdev would be the right thing to do.
21:04:12 efried guess it really just makes sense to make it part of snapshot and have a crisp line
21:13:47 mriedem efried: shelve works for volume-backed and non-volume-backed
21:14:23 efried mriedem: In the compute API I see that volume-backed calls the shelve-offload code path.
21:14:29 mriedem if you shelve but don't offload (which offloading is automatic by default config) then you unshelve within the offload window you do end up on the same host (conductor just starts the server)
21:15:29 efried https://opendev.org/openstack/nova/src/branch/master/nova/compute/api.py#L3867-L3873 ?
21:15:36 mriedem yeah so in the case of volume-backed shelve there is no snapshot image
21:16:02 efried right, but the distinction for 'offload' (at least the one that I care about) is that it calls destroy()
21:16:06 efried whereas non-offload doesn't.
21:16:16 efried uh, unless I got confused.
21:16:34 efried yeah
21:16:47 mriedem yeah shelve without offload powers off the guest and creates a snapshot
21:17:33 efried ack
21:18:17 mriedem i'm drawing a blank on why we shelve offload immediately for volume-backed servers
21:19:19 mriedem just ask one of these guys from 2013 https://review.opendev.org/#/c/34032/
21:19:46 efried was gonna say, seems like it's been that way from the get-go. dansmith co-authored, so...
21:19:51 efried he'll surely know.
21:20:31 mriedem i guess this comment https://review.opendev.org/#/c/34032/21/nova/compute/manager.py@3099
21:20:44 dansmith mriedem: because there's no reason not to offload immediately
21:20:48 efried probably because waaay back then, the distinction with offload was only whether snapshot was necessary, which it isn't for volume (right?)
21:20:50 dansmith mriedem: there's no point in maintaining affinity
21:21:18 dansmith efried: not just no snapshot required, but there's no benefit to unshelve to the same host before offload

Earlier   Later