| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-06 | |||
| 19:32:54 | artom | Nothing else that's interesting for that request ID | |
| 19:33:12 | artom | So... could it be oslo_middleware's sizelimit doing that? How would we make sure? | |
| 19:37:51 | openstackgerrit | Archit Modi proposed openstack/nova stable/queens: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693239 | |
| 19:38:09 | sean-k-mooney | artom: byond a print statemtent in the function im not really sure | |
| 19:38:39 | artom | sean-k-mooney, or I guess removing it from the paste config | |
| 19:38:58 | artom | But paste scares me | |
| 19:39:07 | sean-k-mooney | ya although that might fail somewhere else then | |
| 19:39:43 | mriedem | artom: efried_rollin has dorked around in a lot of that code lately | |
| 19:39:46 | openstackgerrit | Archit Modi proposed openstack/nova stable/rocky: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693240 | |
| 19:39:46 | sean-k-mooney | im sure there is a way to congirue custom midelware to log extra stuff in the 413 case but ya i didnt really want to have to learn that | |
| 19:40:07 | sean-k-mooney | cdent know that code pretty well too i think | |
| 19:40:08 | mriedem | amodi: i see you're doing backports backwards, | |
| 19:40:21 | mriedem | amodi: note that you should be doing train -> stein -> rocky -> queens -> pike | |
| 19:40:22 | mriedem | in that order | |
| 19:40:23 | amodi | mriedem: yes | |
| 19:40:34 | amodi | mriedem: oh i dint know it mattered | |
| 19:40:47 | artom | mriedem, can I blow your mind? | |
| 19:40:50 | mriedem | https://docs.openstack.org/project-team-guide/stable-branches.html#processes | |
| 19:40:52 | artom | amodi is Red Hat QE! | |
| 19:40:52 | amodi | mriedem: im still cherry picking from the master | |
| 19:41:04 | sean-k-mooney | amodi: the patch has to merge on all the newer branches first | |
| 19:41:05 | mriedem | artom: that does not blow my mind | |
| 19:41:10 | sean-k-mooney | amodi: same as downstream | |
| 19:41:10 | artom | He's upstream! | |
| 19:41:40 | mriedem | amodi: the point of that is if we merged the rocky backport, someone got it, and then upgraded to train where the backport isn't merged yet, they'd lose the fix | |
| 19:41:42 | mriedem | and have to re-fix it | |
| 19:41:48 | mriedem | that's why you go latest to oldest | |
| 19:41:54 | amodi | mriedem: i see, yes makes sense | |
| 19:43:04 | melwitt | recently I struggled with locating where an exception was being raised to nova api, all the log said was "INFO [nova.api.openstack.wsgi] HTTP exception thrown: Maximum number of ports exceeded" I'm not sure whether more would be shown at DEBUG | |
| 19:44:01 | melwitt | and for this 413 thing, you said there's zero logging of it.. I can't recall if I've seen that before or how/why it happens | |
| 19:44:55 | artom | melwitt, well, there's just the 1 line | |
| 19:45:12 | artom | And then the request ID appears in a bunch of OPTIONS / requests | |
| 19:45:15 | melwitt | yeah, sorry, I meant other than the POST line with the status code | |
| 19:45:22 | artom | Which I assume are load balancers poking the API | |
| 19:45:49 | sean-k-mooney | artom: no the load blancer would not reuse a request id like that | |
| 19:46:19 | artom | sean-k-mooney, this is queens, did we still have that request ID sharing bug back then? | |
| 19:46:32 | artom | Though that was only for periodic tasks | |
| 19:46:50 | openstackgerrit | Archit Modi proposed openstack/nova stable/train: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693241 | |
| 19:47:14 | openstackgerrit | Archit Modi proposed openstack/nova stable/stein: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693242 | |
| 19:47:30 | sean-k-mooney | ha proxy is dumb and know nothing about the request id. iw we are seeing multipel request with the same request id its form heat/novaclient reusing the request id i woudl suspect | |
| 19:48:10 | sean-k-mooney | its a vm create so it might use the same request id to poll for its completion | |
| 19:48:11 | artom | sean-k-mooney, wait, so heat would be creating the whole stack with a single request ID? | |
| 19:48:12 | artom | Fun | |
| 19:48:14 | mriedem | nobody calls haproxy dumb and gets away with it | |
| 19:48:30 | artom | sean-k-mooney, but... OPTIONS / ? | |
| 19:48:38 | artom | That's obviously health check stuff | |
| 19:48:56 | artom | I've like we've gotten sidelined :) | |
| 19:49:03 | sean-k-mooney | well a http options request is use to discover info about the endpoint | |
| 19:49:15 | mriedem | heh, oh boy, someone just reply-all'ed to 400 people internally | |
| 19:49:59 | sean-k-mooney | yes we may have internal maining lists that people dont understand how to unsubsribe form | |
| 19:50:29 | melwitt | unsubsquirrel | |
| 19:50:36 | sean-k-mooney | and they may or may not like to reply all to them to ask for help to the point ^ is now a thing | |
| 19:51:04 | sean-k-mooney | as is http://www.unsubsquirrel.com/ | |
| 19:52:19 | artom | Oh common | |
| 19:52:31 | mriedem | great now i'm playing that game | |
| 19:52:40 | sean-k-mooney | https://scratch.mit.edu/projects/327361863/ is the beeter version | |
| 19:52:48 | artom | I can't believe the obvious "Unsubscribe you? That's nuts!" pun | |
| 19:52:52 | artom | *they missed | |
| 19:53:28 | melwitt | open a PR | |
| 19:54:52 | artom | Ah no, if you refresh enough times you get it | |
| 19:55:20 | melwitt | oh yeah, I see https://github.com/joellord/subsquirrel/blob/master/index.html#L33 | |
| 20:13:01 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Regression test for bug 1849657 https://review.opendev.org/693248 | |
| 20:13:01 | openstack | bug 1849657 in OpenStack Compute (nova) " allocation key is missing from the binding:profile of the neutron qos port when the server is created by a non-admin user" [Medium,In progress] https://launchpad.net/bugs/1849657 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 20:24:30 | efried | artom: paste? | |
| 20:26:25 | efried | sean-k-mooney: So the one where I added --with-apparmor_profiles and --prefix=/usr failed, but the one where I added that symlink and rebooted succeeded. | |
| 20:27:04 | sean-k-mooney | efried: ok so i need to work on proper apparmor support in a follow up | |
| 20:27:51 | sean-k-mooney | i may have spent the last 3 hour fixing my openstack that i broke but im testing it on a clean vm right now | |
| 20:28:11 | efried | [75800.085251] device br-int entered promiscuous mode | |
| 20:28:11 | efried | sean-k-mooney: the symptoms are all the same, except dmesg in addition to those apparmor="DENIED" messages also shows things like | |
| 20:28:11 | efried | [75800.085042] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length. | |
| 20:28:54 | sean-k-mooney | the other message are normal | |
| 20:29:16 | sean-k-mooney | but you still got the denie message with the profiles comipled in ya | |
| 20:29:46 | efried | well, I'm taking it on faith that "profiles are compiled in". | |
| 20:30:05 | efried | + ./autogen.sh --system --with-apparmor_profiles | |
| 20:30:05 | efried | - ./configure | |
| 20:30:05 | efried | - ./autogen.sh --system | |
| 20:30:05 | efried | and | |
| 20:30:05 | efried | + ./configure --prefix=/usr | |
| 20:30:07 | sean-k-mooney | im not sure if you have to pass that flage to autogen or configure | |
| 20:30:27 | efried | iiuc either one of those should have worked by itself, no? | |
| 20:30:51 | sean-k-mooney | the profies are hardcoded to no | |
| 20:30:55 | efried | because apparmor's snit was with libvirt not being in the right dir? | |
| 20:31:12 | sean-k-mooney | so if you dont explcitly pass the flag they wont be generated | |
| 20:31:31 | sean-k-mooney | it was unhapy with qemu not being in the correct location | |
| 20:31:42 | sean-k-mooney | the --system puts libvirt in the correct place | |
| 20:32:04 | efried | so if I do that symlink thing and reboot, I just have to unstack and restack, not clean, right? | |
| 20:32:18 | sean-k-mooney | yep | |
| 20:32:31 | sean-k-mooney | you dont need to modify the build either | |
| 20:32:45 | sean-k-mooney | if you add the symlink that should be enough with the patch i have pushed up | |
| 20:33:02 | sean-k-mooney | and the local.conf i shared earlier | |
| 20:33:10 | sean-k-mooney | atleast that is what im testing | |
| 20:33:16 | efried | still at ps3 yah? | |
| 20:33:23 | sean-k-mooney | yep | |
| 20:34:01 | sean-k-mooney | i littally created the sysmlink rebooted, cloned devstack, and created the local.conf http://paste.openstack.org/show/785846/ | |
| 20:34:10 | sean-k-mooney | and now im waiting for it to finish stacking | |
| 20:34:18 | sean-k-mooney | i think that should be all that is required | |
| 20:41:50 | efried | what's the incantation to create a VM I can actually ssh into (from the hypervisor, don't care about external stuff) | |
| 20:42:53 | sean-k-mooney | i normally use openstack or virt-manager to do it | |
| 20:43:39 | sean-k-mooney | what os do you want to do it on | |
| 20:44:22 | efried | I mean create via openstack. --network public doesn't seem to be sufficient, or I'm missing something else. Console log shows the big cirros splash, so I think the thing is up. But trying the IPs I see from `server show` is getting me nothing. | |