| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-04 | |||
| 12:52:28 | finucannot | https://review.openstack.org/#/c/564449/1/nova/compute/claims.py | |
| 12:53:34 | finucannot | It also needs to be stored in the RequestSpec object so that we don't need to query neutron from the filters (which we can't do because we don't have correct context) | |
| 12:55:14 | giblet | finucannot: does the RequestSpec loaded from the db in the scheduler or passed via rpc? | |
| 12:55:31 | finucannot | The former, to the best of my knowledge | |
| 12:56:00 | giblet | finucannot: then I agree that you have to store the NetworkREquestList to the db along with the RequestSpec | |
| 12:56:43 | finucannot | Yeah, I mapped the whole thing here http://paste.openstack.org/show/720365/ | |
| 12:59:58 | giblet | finucannot: I think the RequestSpec is passed to the scheduler via rpc https://github.com/openstack/nova/blob/5d97937c3c56a3e240a3350a7a9f0e3dcb954c52/nova/scheduler/rpcapi.py#L132 | |
| 13:00:34 | giblet | the spec_obj there is a RequestSpec obj | |
| 13:03:33 | giblet | finucannot: also the build_and_run_instance leading to the claim gets the instance object via rpc | |
| 13:04:46 | finucannot | Yup, so if I wanted to, for example, pass an additional 'network_requests' parameter to 'claim()', I guess I'd have to bump the RPC version | |
| 13:05:01 | finucannot | I'm pretty sure I looked at that though and it wasn't possible. Lemme look again | |
| 13:06:29 | giblet | finucannot: you can still add the network_requests as a field to the Instance ovo and to the RequestSpec ovo, but you not necessary to persist the content of that fields to the db, as the user of that field always get the object via rpc and the sender can regenerate the content of the network_request | |
| 13:07:44 | giblet | so in case of boot, the conductor generates the content of the network_request field and pass it down via the ReqestSpec of the Instance object | |
| 13:07:52 | giblet | similarly in case of a VM move operation | |
| 13:09:49 | finucannot | Yeah, that shouldn't be an issue for RequestSpec as I don't think those are persisted. What about Instance though. Can you mark a field in a persistent object as non-persistent? | |
| 13:09:49 | giblet | persistence is itching my mind because this data is already persisted in neutron so as soon as nova also persist it we will have two possible divergent copies | |
| 13:11:10 | giblet | finucannot: there is a request_specs table in the api db so your technical problem how to not persist a field is valid for both object | |
| 13:13:17 | giblet | finucannot: we should ask dansmith about this persistency issue but I have an idea | |
| 13:13:24 | giblet | finucannot: make the new field lazy_loaded | |
| 13:13:49 | giblet | finucannot: and try to generate the value of the field in obj_load_attr | |
| 13:14:41 | giblet | finucannot: or another idea. never set the field in _from_db_object but set it from the conductor when the data is available | |
| 13:15:48 | giblet | finucannot: also never pass this field to the db method in save() | |
| 13:16:57 | jmccarthy | @jaypipes You about ? | |
| 13:18:03 | finucannot | giblet: They're all good ideas. Let me explore them and see what I can do | |
| 13:18:16 | finucannot | Agreed on split brain issue too. | |
| 13:18:57 | giblet | finucannot: OK. I will have same issue with the bandwidth related resource requests coming from the neutron port and going to placement a_c query via nova-conductor and nova-scheduler | |
| 13:20:06 | giblet | finucannot: but you are further down the road as you already have code up | |
| 13:20:49 | finucannot | Yeah, I found I needed to write that so I could actually reason about stuff properly, otherwise I was guessing a lot of it :D | |
| 13:21:01 | finucannot | Turns out I didn't know the scheduler all that well, heh | |
| 13:24:05 | giblet | finucannot: yeah devil is in the details and in this cases writing code means seeing the details | |
| 13:26:10 | mriedem | bauwser: can you get https://review.openstack.org/#/c/566161/ ? | |
| 13:28:04 | pipesinpain | jmccarthy: yes | |
| 13:28:30 | pipesinpain | finucannot: cool, will review this morning. | |
| 13:28:49 | bauwser | mriedem: ack, looking | |
| 13:30:06 | jmccarthy | pipesinpain: Hiya ! Just wondering if you might have some insight into this bug here ? 1769131 It sounds a lot like 1666831 | |
| 13:31:21 | giblet | mriedem: both pipesinpain, fried_rice and mlavalle seems to be OK with the nova bandwidth spec. So if you are interested then this is a good time to look at it https://review.openstack.org/#/c/502306/ | |
| 13:31:50 | fried_rice | pipesinpain: elbow? | |
| 13:32:47 | pipesinpain | fried_rice: yeah :( | |
| 13:33:04 | pipesinpain | jmccarthy: will take a look after caffeinating. | |
| 13:33:07 | fried_rice | Did you golf or something? | |
| 13:33:17 | pipesinpain | fried_rice: nope. didn't do anything. | |
| 13:33:39 | fried_rice | boo. Your elbow and my knees. No idea what happened, just swelling and pain this morning. Hell to get old. | |
| 13:35:14 | giblet | pipesinpain, fried_rice: I hope both of you get better soon | |
| 13:35:21 | fried_rice | thanks giblet | |
| 13:35:56 | pipesinpain | giblet: thx man. | |
| 13:36:33 | jmccarthy | Erg vpn blipped | |
| 13:37:25 | jmccarthy | I've been looking at driver.py but I may be misinterpreting some of the comments that are in there - this seems relevant https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1164 | |
| 13:37:25 | mriedem | giblet: ok will do | |
| 13:37:44 | giblet | mriedem: thanks | |
| 13:38:31 | pipesinpain | jmccarthy: you are using volume-backed instances, yes? | |
| 13:38:41 | pipesinpain | jmccarthy: i.e. boot-from-volume | |
| 13:39:44 | jmccarthy | I have to check what horizon says, spawning cirros with cinder, so volume backed yep | |
| 13:40:11 | pipesinpain | jmccarthy: right. so that is the behaviour by design. | |
| 13:40:38 | pipesinpain | jmccarthy: maciej had added that code to ensure that the volume wasn't removed when resizing. | |
| 13:40:57 | pipesinpain | jmccarthy: for boot-from-volume instances, you don't want to delete the original root volume, clearly :) | |
| 13:41:28 | pipesinpain | jmccarthy: I'm wondering if this is because of some thin/sparse copy-on-write stuff that Docker is doing maybe.. | |
| 13:42:05 | jmccarthy | pipesinpain: The volume itself is in cinder and seems safe enough in my case ? It's this disk.info that shows up on host that is throwing me off | |
| 13:42:22 | pipesinpain | jmccarthy: but in all honesty, I'm pretty much the worst person to ask on this :) a) I have little knowledge of the block device layer, b) I don't use (or support) boot-from-volume, and c) I don't use resize ;) | |
| 13:42:55 | pipesinpain | jmccarthy: good people to hit up are mriedem when he's online and maybe mdbooth | |
| 13:43:08 | jmccarthy | pipesinpain: Oh ok I thought you might know, since my disk.info issue sounds a lot like a bug you fixed up | |
| 13:43:43 | jmccarthy | pippesinpain: Ok I'll ask around - I'm just not sure why it would leave this file on the host afterwards, seems odd | |
| 13:43:44 | pipesinpain | jmccarthy: that was more melwitt (soon to be jgwentworth) that fixed that bug :) | |
| 13:43:48 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Remove support for monitoring Intel CMT `perf` events https://review.openstack.org/565242 | |
| 13:43:57 | jmccarthy | pipesinpain: Ok ! :) Sorry for the noise | |
| 13:44:05 | pipesinpain | jmccarthy: no problem at all, man :) | |
| 13:44:38 | jmccarthy | Any of those people sitting quielty by ? ;) | |
| 13:46:14 | pipesinpain | jmccarthy: melwitt will be up in the next hour or so. mriedem will be by soon I imagine (PST and CST respectively) | |
| 13:46:36 | jmccarthy | In horizon, if it boots from image and you create a new volume at the same time in persistent storage for it, that not boot-from-volume strictly speaking, correct ? | |
| 13:46:44 | pipesinpain | jmccarthy: mdbooth is in GMT so should be around (he's likely seen "boot from volume" and run screaming for the hills) :P | |
| 13:47:04 | pipesinpain | jmccarthy: no, that's not boot-from-volume. | |
| 13:47:26 | jmccarthy | pipesinpain: That is a volume backed instance, but booted off image - does that have a special name ? | |
| 13:47:35 | pipesinpain | jmccarthy: boot-from-volume is when you supply a prebuilt bootable volume as your root disk (disk 0) for the instance | |
| 13:47:50 | pipesinpain | jmccarthy: that's a normal instance, then | |
| 13:47:59 | jmccarthy | pipesinpain: I am using volumes, but nope, not boot from vol, ok normal instances | |
| 13:48:04 | pipesinpain | jmccarthy: you mean you just added a non-bootable data volume to the instance, right? | |
| 13:48:17 | wolverineav | hi, while working on a neutron plugin, i ran into a weird situation where nova sends multiple requests for bind_port on different hosts, before neutron has had a chance to complete the first request. i’m guessing this would be by design, since I don’t see error or warning about the first request failing. in case anyone would know where the code path for this resides, that’d be great! | |
| 13:48:24 | pipesinpain | jmccarthy: good man, yes, that's not boot-from-volume :) | |
| 13:48:54 | jmccarthy | pipesinpain: Well not manually ? Horizon just does it .. "Instance source is the template used to create an instance. You can use an image, a snapshot of an instance (image snapshot), a volume or a volume snapshot (if enabled). You can also choose to use persistent storage by creating a new volume." | |
| 13:49:04 | pipesinpain | wolverineav: ironic? | |
| 13:49:26 | wolverineav | no, instance creation in overcloud | |
| 13:49:30 | mriedem | jmccarthy: nova show the instance on the CLI, if the image_ref is "", it's volume-backed (boot from volume) | |
| 13:49:37 | pipesinpain | jmccarthy: right. and you chose the "persistent storage by creating a new volume", yes? | |
| 13:49:50 | wolverineav | pipesinpain: no, instance creation in overcloud | |
| 13:50:02 | jmccarthy | pipesinpain: Mine just says 'create new volume' but yep | |
| 13:50:54 | pipesinpain | wolverineav: multiple requests to bind the port to different hosts for the same instance? :( | |
| 13:51:12 | jmccarthy | pipesinpain: Afaik this creates an instance the lives solely in the volume like, and subsequently will boot from there | |
| 13:52:15 | jmccarthy | pipesinpain: 'pipesinpain' sounds painful | |
| 13:56:47 | pipesinpain | jmccarthy: no. unless I'm mistaken (very much a possibility), "create a volume" just means the "You can also choose to use persistent storage by creating a new volume." option. In other words, it creates a data volume (non-bootable) for the instance to use. | |
| 13:56:58 | wolverineav | pipesinpain: sorry about the phrasing - i'm looking at the problem completely from neutron logs - i (neutron) get port_update request with 2 different hosts in a span of less than 30secs. typically, there's about 3 port updates until its finally bound and in active state. i couldn't figure out where the aggresive timeout is, that forces a retry to another host. so i was checking in nova group if that was known :) | |
| 13:57:46 | pipesinpain | wolverineav: but this is a port binding request for the *same* instance, yes? | |
| 13:58:11 | wolverineav | pipesinpain: yes, correct. | |
| 13:58:49 | pipesinpain | wolverineav: that is *seriously* odd... | |
| 13:59:07 | pipesinpain | wolverineav: I can't understand why multiple hosts would be port-bound to the same instance at the same time | |
| 13:59:28 | pipesinpain | wolverineav: that would mean that the scheduler essentially picked multiple hosts for the same instance. | |
| 13:59:51 | pipesinpain | wolverineav: or that a retry is occurring nearly instantaneously. | |
| 14:00:31 | pipesinpain | wolverineav: since the scheduler absolutely cannot simultaneously pick multiple hosts for the same instance, it must be related to the retry. | |
| 14:00:49 | jmccarthy | pipesinpain: Hmm ok, what would that instance be booting from for future boots like after it's created ? I thought it did stuff with the image and used it to make the vol bootable like. I only see a /dev/sda and /dev/sda1 in the instance itself (which I thought was the new volume it created, made bootable) - I could be missing something .. | |
| 14:01:01 | pipesinpain | wolverineav: can you please look in the compute logs for "Retry"? | |
| 14:01:27 | pipesinpain | jmccarthy: it would be booting from the same original image you selected in horizon. | |
| 14:01:37 | pipesinpain | jmccarthy: and would *attach* the persistent data volume after boot. | |