| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-04 | |||
| 13:13:17 | giblet | finucannot: we should ask dansmith about this persistency issue but I have an idea | |
| 13:13:24 | giblet | finucannot: make the new field lazy_loaded | |
| 13:13:49 | giblet | finucannot: and try to generate the value of the field in obj_load_attr | |
| 13:14:41 | giblet | finucannot: or another idea. never set the field in _from_db_object but set it from the conductor when the data is available | |
| 13:15:48 | giblet | finucannot: also never pass this field to the db method in save() | |
| 13:16:57 | jmccarthy | @jaypipes You about ? | |
| 13:18:03 | finucannot | giblet: They're all good ideas. Let me explore them and see what I can do | |
| 13:18:16 | finucannot | Agreed on split brain issue too. | |
| 13:18:57 | giblet | finucannot: OK. I will have same issue with the bandwidth related resource requests coming from the neutron port and going to placement a_c query via nova-conductor and nova-scheduler | |
| 13:20:06 | giblet | finucannot: but you are further down the road as you already have code up | |
| 13:20:49 | finucannot | Yeah, I found I needed to write that so I could actually reason about stuff properly, otherwise I was guessing a lot of it :D | |
| 13:21:01 | finucannot | Turns out I didn't know the scheduler all that well, heh | |
| 13:24:05 | giblet | finucannot: yeah devil is in the details and in this cases writing code means seeing the details | |
| 13:26:10 | mriedem | bauwser: can you get https://review.openstack.org/#/c/566161/ ? | |
| 13:28:04 | pipesinpain | jmccarthy: yes | |
| 13:28:30 | pipesinpain | finucannot: cool, will review this morning. | |
| 13:28:49 | bauwser | mriedem: ack, looking | |
| 13:30:06 | jmccarthy | pipesinpain: Hiya ! Just wondering if you might have some insight into this bug here ? 1769131 It sounds a lot like 1666831 | |
| 13:31:21 | giblet | mriedem: both pipesinpain, fried_rice and mlavalle seems to be OK with the nova bandwidth spec. So if you are interested then this is a good time to look at it https://review.openstack.org/#/c/502306/ | |
| 13:31:50 | fried_rice | pipesinpain: elbow? | |
| 13:32:47 | pipesinpain | fried_rice: yeah :( | |
| 13:33:04 | pipesinpain | jmccarthy: will take a look after caffeinating. | |
| 13:33:07 | fried_rice | Did you golf or something? | |
| 13:33:17 | pipesinpain | fried_rice: nope. didn't do anything. | |
| 13:33:39 | fried_rice | boo. Your elbow and my knees. No idea what happened, just swelling and pain this morning. Hell to get old. | |
| 13:35:14 | giblet | pipesinpain, fried_rice: I hope both of you get better soon | |
| 13:35:21 | fried_rice | thanks giblet | |
| 13:35:56 | pipesinpain | giblet: thx man. | |
| 13:36:33 | jmccarthy | Erg vpn blipped | |
| 13:37:25 | jmccarthy | I've been looking at driver.py but I may be misinterpreting some of the comments that are in there - this seems relevant https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1164 | |
| 13:37:25 | mriedem | giblet: ok will do | |
| 13:37:44 | giblet | mriedem: thanks | |
| 13:38:31 | pipesinpain | jmccarthy: you are using volume-backed instances, yes? | |
| 13:38:41 | pipesinpain | jmccarthy: i.e. boot-from-volume | |
| 13:39:44 | jmccarthy | I have to check what horizon says, spawning cirros with cinder, so volume backed yep | |
| 13:40:11 | pipesinpain | jmccarthy: right. so that is the behaviour by design. | |
| 13:40:38 | pipesinpain | jmccarthy: maciej had added that code to ensure that the volume wasn't removed when resizing. | |
| 13:40:57 | pipesinpain | jmccarthy: for boot-from-volume instances, you don't want to delete the original root volume, clearly :) | |
| 13:41:28 | pipesinpain | jmccarthy: I'm wondering if this is because of some thin/sparse copy-on-write stuff that Docker is doing maybe.. | |
| 13:42:05 | jmccarthy | pipesinpain: The volume itself is in cinder and seems safe enough in my case ? It's this disk.info that shows up on host that is throwing me off | |
| 13:42:22 | pipesinpain | jmccarthy: but in all honesty, I'm pretty much the worst person to ask on this :) a) I have little knowledge of the block device layer, b) I don't use (or support) boot-from-volume, and c) I don't use resize ;) | |
| 13:42:55 | pipesinpain | jmccarthy: good people to hit up are mriedem when he's online and maybe mdbooth | |
| 13:43:08 | jmccarthy | pipesinpain: Oh ok I thought you might know, since my disk.info issue sounds a lot like a bug you fixed up | |
| 13:43:43 | jmccarthy | pippesinpain: Ok I'll ask around - I'm just not sure why it would leave this file on the host afterwards, seems odd | |
| 13:43:44 | pipesinpain | jmccarthy: that was more melwitt (soon to be jgwentworth) that fixed that bug :) | |
| 13:43:48 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Remove support for monitoring Intel CMT `perf` events https://review.openstack.org/565242 | |
| 13:43:57 | jmccarthy | pipesinpain: Ok ! :) Sorry for the noise | |
| 13:44:05 | pipesinpain | jmccarthy: no problem at all, man :) | |
| 13:44:38 | jmccarthy | Any of those people sitting quielty by ? ;) | |
| 13:46:14 | pipesinpain | jmccarthy: melwitt will be up in the next hour or so. mriedem will be by soon I imagine (PST and CST respectively) | |
| 13:46:36 | jmccarthy | In horizon, if it boots from image and you create a new volume at the same time in persistent storage for it, that not boot-from-volume strictly speaking, correct ? | |
| 13:46:44 | pipesinpain | jmccarthy: mdbooth is in GMT so should be around (he's likely seen "boot from volume" and run screaming for the hills) :P | |
| 13:47:04 | pipesinpain | jmccarthy: no, that's not boot-from-volume. | |
| 13:47:26 | jmccarthy | pipesinpain: That is a volume backed instance, but booted off image - does that have a special name ? | |
| 13:47:35 | pipesinpain | jmccarthy: boot-from-volume is when you supply a prebuilt bootable volume as your root disk (disk 0) for the instance | |
| 13:47:50 | pipesinpain | jmccarthy: that's a normal instance, then | |
| 13:47:59 | jmccarthy | pipesinpain: I am using volumes, but nope, not boot from vol, ok normal instances | |
| 13:48:04 | pipesinpain | jmccarthy: you mean you just added a non-bootable data volume to the instance, right? | |
| 13:48:17 | wolverineav | hi, while working on a neutron plugin, i ran into a weird situation where nova sends multiple requests for bind_port on different hosts, before neutron has had a chance to complete the first request. i’m guessing this would be by design, since I don’t see error or warning about the first request failing. in case anyone would know where the code path for this resides, that’d be great! | |
| 13:48:24 | pipesinpain | jmccarthy: good man, yes, that's not boot-from-volume :) | |
| 13:48:54 | jmccarthy | pipesinpain: Well not manually ? Horizon just does it .. "Instance source is the template used to create an instance. You can use an image, a snapshot of an instance (image snapshot), a volume or a volume snapshot (if enabled). You can also choose to use persistent storage by creating a new volume." | |
| 13:49:04 | pipesinpain | wolverineav: ironic? | |
| 13:49:26 | wolverineav | no, instance creation in overcloud | |
| 13:49:30 | mriedem | jmccarthy: nova show the instance on the CLI, if the image_ref is "", it's volume-backed (boot from volume) | |
| 13:49:37 | pipesinpain | jmccarthy: right. and you chose the "persistent storage by creating a new volume", yes? | |
| 13:49:50 | wolverineav | pipesinpain: no, instance creation in overcloud | |
| 13:50:02 | jmccarthy | pipesinpain: Mine just says 'create new volume' but yep | |
| 13:50:54 | pipesinpain | wolverineav: multiple requests to bind the port to different hosts for the same instance? :( | |
| 13:51:12 | jmccarthy | pipesinpain: Afaik this creates an instance the lives solely in the volume like, and subsequently will boot from there | |
| 13:52:15 | jmccarthy | pipesinpain: 'pipesinpain' sounds painful | |
| 13:56:47 | pipesinpain | jmccarthy: no. unless I'm mistaken (very much a possibility), "create a volume" just means the "You can also choose to use persistent storage by creating a new volume." option. In other words, it creates a data volume (non-bootable) for the instance to use. | |
| 13:56:58 | wolverineav | pipesinpain: sorry about the phrasing - i'm looking at the problem completely from neutron logs - i (neutron) get port_update request with 2 different hosts in a span of less than 30secs. typically, there's about 3 port updates until its finally bound and in active state. i couldn't figure out where the aggresive timeout is, that forces a retry to another host. so i was checking in nova group if that was known :) | |
| 13:57:46 | pipesinpain | wolverineav: but this is a port binding request for the *same* instance, yes? | |
| 13:58:11 | wolverineav | pipesinpain: yes, correct. | |
| 13:58:49 | pipesinpain | wolverineav: that is *seriously* odd... | |
| 13:59:07 | pipesinpain | wolverineav: I can't understand why multiple hosts would be port-bound to the same instance at the same time | |
| 13:59:28 | pipesinpain | wolverineav: that would mean that the scheduler essentially picked multiple hosts for the same instance. | |
| 13:59:51 | pipesinpain | wolverineav: or that a retry is occurring nearly instantaneously. | |
| 14:00:31 | pipesinpain | wolverineav: since the scheduler absolutely cannot simultaneously pick multiple hosts for the same instance, it must be related to the retry. | |
| 14:00:49 | jmccarthy | pipesinpain: Hmm ok, what would that instance be booting from for future boots like after it's created ? I thought it did stuff with the image and used it to make the vol bootable like. I only see a /dev/sda and /dev/sda1 in the instance itself (which I thought was the new volume it created, made bootable) - I could be missing something .. | |
| 14:01:01 | pipesinpain | wolverineav: can you please look in the compute logs for "Retry"? | |
| 14:01:27 | pipesinpain | jmccarthy: it would be booting from the same original image you selected in horizon. | |
| 14:01:37 | pipesinpain | jmccarthy: and would *attach* the persistent data volume after boot. | |
| 14:01:38 | wolverineav | pipesinpain: i had to follow threads to figure out what was happening. it caused some inconsistency issue between neutron and DB and backend network controller when two requests got processed in different order in neutron vs calls made to controller. | |
| 14:02:06 | wolverineav | pipesinpain: yes, let me check. I did not notice anything earlier. but i'll enable debug to be sure | |
| 14:02:13 | jmccarthy | pipesinpain: Hmm ok that's still ok, I mean there is no other disk file for it in /var/lib/instances anywhere (at least that I could find) | |
| 14:02:23 | pipesinpain | wolverineav: any chance you can upload the compute logs somewhere? or at least, a section of them around where this is happening? | |
| 14:02:59 | jmccarthy | pipesinpain: (of whatever the path is, I'm sure I typed that wrong)# | |
| 14:03:40 | wolverineav | pipesinpain: it doesn't always happen, so i don't have debug enabled logs (first happened at a customer site). i'll ping back here (/mailing list) once i have that reproduced. | |
| 14:04:32 | pipesinpain | wolverineav: ++, thank you sir! :) | |
| 14:06:38 | jmccarthy | pipesinpain: Sorry yes of course - I am confusing the preparation of volume to be used to boot from - nevermind :) I still think there is a bug or something strange going on :) Thanks for your help ! | |
| 14:10:08 | mriedem | giblet: weren't we at one point talking about the scheduler creating the allocations against the nic bw providers? now that gets passed down to neutron through port binding in the latest revision of the spec. | |
| 14:10:17 | pipesinpain | jmccarthy: no problemo :) | |
| 14:10:19 | mriedem | honestly i can't remember which one i was pushing for a month ago | |
| 14:10:50 | giblet | mriedem: scheduler does the claim, but nova sends to neutron what was claimed | |
| 14:11:02 | mriedem | oh, ok, that was'nt clear to me yet | |
| 14:13:28 | giblet | mriedem: please leave a comment and I will clarify that in the spec | |
| 14:13:30 | Cardoe | mriedem: You asked about rebooting rescued instances the other day at Rackspace here. I'm trying to find you an answer. | |
| 14:13:54 | mriedem | Cardoe: that would be great, thanks - also posted to the openstack-dev ML | |
| 14:14:19 | Cardoe | full disclosure, I'm fairly new here and I don't have any OpenStack experience/knowledge. | |