| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-11 | |||
| 22:09:45 | dansmith | efried: I understand that we avoid asking for the new thing in a roundabout way by not exposing a trait, cyborg not exposing inventory, placement not returning any candidates, and thus us not actually running our control plane code, | |
| 22:09:46 | efried | Sundar: If cyborg agent is shiny and new on a host, but the nova-compute on the host is downlevel | |
| 22:09:47 | efried | and then you upgrade the nova-compute | |
| 22:09:47 | efried | will cyborg eventually figure that out and realize it should now start reporting accel inventory on that host? | |
| 22:10:03 | dansmith | efried: but it would be a lot more in line with our regular checks to just be explicit about it, even if we should never hit it because of the other | |
| 22:10:21 | efried | dansmith: okay, so what is it that you're suggesting exactly? I didn't pick up on that. | |
| 22:10:50 | efried | An RPC version check from conductor to compute? | |
| 22:10:51 | openstackgerrit | Mykola Yakovliev proposed openstack/nova master: Fix boot_roles in InstanceSystemMetadata https://review.opendev.org/698040 | |
| 22:11:04 | dansmith | efried: if you pass the ARQs down in the spawn call, you need an RPC version, so you get a service version, which is something you can just check for easily in conductor, alternately just bump the service version without the rpc and check for that in conductor | |
| 22:11:23 | dansmith | efried: or expose it as a trait and have conductor/scheduler filter or check for that trait before agreeing to use that compute | |
| 22:11:55 | dansmith | efried: perhaps the easiest thing would be just "if we have accel resources, also add CAN_DO_CYBORG_SHIT trait into the requirements" before we call placement during scheduling? | |
| 22:12:05 | dansmith | you'll need to do it again for attach when we add that | |
| 22:12:14 | dansmith | so CAN_DO_CYBORG_SHIT_V2alpha | |
| 22:13:02 | efried | dansmith: I'm not offended by the thought of those capability traits, which seem on par with what we have for e.g. "can do multiattach" or whatever | |
| 22:13:09 | efried | so IMO we should do those regardless | |
| 22:13:12 | dansmith | ack | |
| 22:13:17 | dansmith | so cyborg can look for that, | |
| 22:13:21 | efried | those are simply flags on that virt driver dict | |
| 22:13:25 | dansmith | and we throw that into the scheduling request also if we're doing accel stuff? | |
| 22:13:49 | efried | Well, I don't think we need it in the sched request, if cyborg is using it to decide whether to present the inventory in the first place. | |
| 22:14:09 | efried | but if you think it should be there for form's sake, I wouldn't object. | |
| 22:14:11 | dansmith | again, that puts the onus on cyborg to be responsible, which I don't like | |
| 22:14:19 | efried | where's the trust?? | |
| 22:14:29 | Sundar | So, the use case for all this is that, the operator added accelerator to a compute node but did not upgrade n-cpu on that node? | |
| 22:14:31 | dansmith | right, thank you :) | |
| 22:14:35 | dansmith | efried: let me check my butt, hang on | |
| 22:14:57 | efried | That's not spelled "onus" dansmith | |
| 22:15:06 | dansmith | hehe | |
| 22:15:08 | efried | Sundar: yes, exactly. | |
| 22:15:37 | efried | Sundar: we're trying to make nova account for poorly-behaved a) operators, b) service code from other services. | |
| 22:15:57 | efried | and not-work in a predictable rather than unpredictable (and potentially unrecoverable) way. | |
| 22:16:11 | dansmith | just to be clear, | |
| 22:16:33 | dansmith | I definitely think that the rpc method change is the right thing to do, instead of just coding in new assumptions on both sides | |
| 22:16:57 | dansmith | I understand the hesitation to that, so I won't hold strictly to it, despite it being the safer option, IMHO | |
| 22:17:28 | efried | dansmith: if we did that, we would have to have a post-placement filter to remove hosts based on RPC version, right? | |
| 22:17:35 | efried | perhaps we already have that. | |
| 22:17:47 | Sundar | What's wrong with the option where Nova places a CAN_DO_CYBORG trait on some compute node RPs, and factors that into Placement query? | |
| 22:18:14 | efried | Sundar: nothing's wrong with that, and we should do it, but we're talking about doing another thing as well. | |
| 22:18:22 | dansmith | efried: no, you'd still want to pick a capable host during scheduling, and scheduling based on rpc version is not a thing | |
| 22:18:39 | efried | dansmith: so what's the RPC version for again? Just passing the arq uuids? | |
| 22:19:37 | dansmith | efried: yes, and to crystalize our request to the compute node so that (a) the RPC layer can tell if we're (for whatever reason) asking the compute node to do something it can't handle, and so that compute doesn't bake the assumption of the instance having a magical key in its flavor into meaning that it should call to cyborg to do all this stuff | |
| 22:20:13 | dansmith | efried: because sometime in the future, the mere presence of that flavor key will imply a slightly different thing, and we will have an upgrade concern to deal with because we have a bunch of old computes that will still be doing the assume-it thing | |
| 22:20:41 | dansmith | I made reference to this in a previous discussion about all of this | |
| 22:22:16 | efried | This is getting into details I probably don't care about rn, but do we get (a) for free (does the call bounce if the compute is too old) or do we have to do an explicit check? | |
| 22:22:37 | dansmith | summarized the bits you do care about here: https://review.opendev.org/#/c/631244/51/nova/conductor/manager.py | |
| 22:23:30 | efried | ack | |
| 22:23:55 | dansmith | efried: the rpc layer knows what the minimum supported version is across the cluster, so when it is converting the request, it can explode if doing so would drop an arqs parameter | |
| 22:24:14 | dansmith | efried: it gets you the circuit breaker, not necessarily the "graceful" part -- that should still be scheduler-based | |
| 22:24:31 | efried | cool | |
| 22:26:28 | dansmith | Sundar: while you're here, what's the plan for getting some small example of this running with an actual pci device in the intel pci ci system (or something) ? | |
| 22:27:53 | Sundar | dansmith: We have the server and the FPGA in a lab, and one person is working on that. We expect it by Jan, if not this year. | |
| 22:28:52 | Sundar | FWIW, I check each patch set with real FPGAs, with different device profiles, and also the fake driver. | |
| 22:29:49 | efried | We should have a 3rd party CI called "Intel Cyborg Sundar Manual CI" | |
| 22:30:08 | dansmith | Sundar: that would be good except that I totally don't trust you as much as a computer | |
| 22:30:10 | dansmith | :) | |
| 22:30:10 | Sundar | We already have that hehe | |
| 22:30:35 | dansmith | Sundar: so one server and one fpga and one person.. what is the plan? to have that be a manual "recheck use fpgas" command or something? | |
| 22:30:41 | Sundar | dansmith: Too bad, I am more reliable than other past Intel CIs ;) | |
| 22:30:56 | dansmith | Sundar: that's true, although doesn't really change my feelings about it :) | |
| 22:31:11 | efried | dansmith: experimental queue? | |
| 22:31:33 | dansmith | efried: that's what I meant.. some not-on-everything per-request method | |
| 22:31:37 | efried | or severely restrict files | |
| 22:31:53 | efried | but yeah | |
| 22:32:03 | dansmith | per-request would be better I think, if it's severely constrained | |
| 23:09:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug cross-cell resize https://review.opendev.org/698304 | |
| 23:17:18 | openstackgerrit | Merged openstack/nova master: Add resource provider allocation unset example to troubleshooting doc https://review.opendev.org/696582 | |
| 23:17:25 | openstackgerrit | Merged openstack/nova master: nova-net: Convert remaining API tests to use neutron https://review.opendev.org/696509 | |
| 23:38:17 | pmatulis | how do i map the hypervisor long ID (server show) to the hypervisor hostname (hypervisor list/show)? | |
| 23:40:05 | dansmith | pmatulis: if you server-show as admin, you should see the unobfuscated hostid , IIRC | |
| 23:41:56 | dansmith | pmatulis: as OS-EXT-SRV-ATTR:hypervisor_hostname | |
| #openstack-nova - 2019-12-12 | |||
| 00:01:04 | pmatulis | dansmith, yeah, i'm going to change a nova policy to make that available to non-admins | |
| 00:16:24 | efried | dansmith: here's a weird one: If I shelve my instance, and then my compute goes down, would a subsequent offload "succeed"? | |
| 00:16:24 | efried | The virt driver's destroy() would have to happen when the compute came back up later, just like we reap instances that are deleted when their compute is down. | |
| 00:16:24 | efried | But do we do that, or do we just fail the offload and ERROR the instance? | |
| 00:28:52 | dansmith | efried: offload should succeed when it comes back up, but would fail while the compute is down with just an rpc failure type thing | |
| 00:29:10 | efried | okay. thanks. | |
| 00:29:22 | dansmith | O dpm | |
| 00:29:24 | dansmith | wow. | |
| 00:29:31 | dansmith | I don't think there's a local delete sort of thing for offload | |
| 00:29:51 | dansmith | yeah, there's not | |
| 00:30:00 | efried | okay. | |
| 00:30:21 | efried | dansmith: typos courtesy of "shift right"? | |
| 00:30:32 | dansmith | yep | |
| 00:30:52 | dansmith | efried: also, shelve_offload() is a cast, so no error result or moving instance to ERROR if it fails due to a down compute | |
| 00:30:52 | efried | I use dvorak | |
| 00:31:03 | efried | so if I shift right, I just look like I'm having a stroke. | |
| 00:31:16 | dansmith | heh | |
| 00:31:31 | efried | dansmith: that was my next question: offload kinda happens "later" sometimes, right? | |
| 00:31:46 | efried | so... does it still have the context (auth etc) of the user who did the original shelve? | |
| 00:32:00 | dansmith | it can be requested via the API, or via periodic after a timeout yeah | |
| 00:32:11 | dansmith | efried: no it could be the operator via the API, or no context via periodic | |
| 00:32:32 | efried | ugh | |
| 00:32:53 | dansmith | in fact, I can't think of any reason a regular user would ever manually offload | |
| 00:33:09 | efried | make space? | |
| 00:33:26 | efried | but anyway, it's another monkey wrench for me. | |
| 00:33:27 | dansmith | shelve is what causes them to start saving money, offload is the cache purge for the operator so something else can go there, but shouldn't really impact the user, other than to make unshelve take longer | |
| 00:33:41 | dansmith | right, the user doesn't care to make space, they're already saving the money | |
| 00:33:51 | efried | oh, yeah, I guess the space is reclaimed at shelve time. | |
| 00:33:58 | dansmith | the operator might offload a shelved instance to get the space back before the timer | |
| 00:34:09 | dansmith | well, not really, | |